Solaris Performance and Tools: Dtrace and Mdb Techniques for Solaris 10 and Opensolaris by Richard McDougall, Jim Mauro, Brendan Gregg

From the Publisher

Solaris Performance and Tools provides comprehensive coverage of the powerful utilities bundled with Solaris 10 and OpenSolaris, including the Solaris Dynamic Tracing facility, DTrace, and the Modular Debugger, MDB. It provides a systematic approach to understanding performance and behavior, including:

• Analyzing CPU utilization by the kernel and applications, including reading and understanding hardware counters
• Process-level resource usage and profiling
• Disk IO behavior and analysis
• Memory usage at the system and application level
• Network performance
• Monitoring and profiling the kernel, and gathering kernel statistics
• Using DTrace providers and aggregations
• MDB commands and a complete MDB tutorial

Product Details

• Table of Contents
• Forewords & Introductions

Table of Contents

Forewords & Introductions

Welcome to the second edition of Solaris™ Internals and its companion volume, Solaris™ Performance and Tools. It has been almost five years since the release of the first edition, during which time we have had the opportunity to communicate with a great many Solaris users, software developers, system administrators, database administrators, performance analysts, and even the occasional kernel hacker. We are grateful for all the feedback, and we have made specific changes to the format and content of this edition based on reader input. Read on to learn what is different. We look forward to continued communication with the Solaris community.

These books are about the internals of Sun’s Solaris Operating System—specifically, the SunOS kernel. Other components of Solaris, such as windowing systems for desktops, are not covered. The first edition of Solaris™ Internals covered Solaris releases 2.5.1, 2.6, and Solaris 7. These volumes focus on Solaris 10, with updated information for Solaris 8 and 9.

In the first edition, we wanted not only to describe the internal components that make the Solaris kernel tick, but also to provide guidance on putting the information to practical use. These same goals apply to this work, with further emphasis on the use of bundled (and in some cases unbundled) tools and utilities that can be used to examine and probe a running system. Our ability to illustrate more of the kernel’s inner workings with observability tools is facilitated in no small part by the inclusion of some revolutionary and innovative technology in Solaris 10—DTrace, a dynamic kernel tracingframework. DTrace is one of many new technologies in Solaris 10, and is used extensively throughout this text.

In working on the second edition, we enlisted the help of several friends and colleagues, many of whom are part of Solaris kernel engineering. Their expertise and guidance contributed significantly to the quality and content of these books. We also found ourselves expanding topics along the way, demonstrating the use of dtrace(1), mdb(1), kstat(1), and other bundled tools. So much so that we decided early on that some specific coverage of these tools was necessary, and chapters were written to provide readers with the required background information on the tools and utilities. From this, an entire chapter on using the tools for performance and behavior analysis evolved.

As we neared completion of the work, and began building the entire manuscript, we ran into a bit of a problem—the size. The book had grown to over 1,500 pages. This, we discovered, presented some problems in the publishing and production of the book. After some discussion with the publisher, it was decided we should break the work up into two volumes.

Solaris™ Internals. This represents an update to the first edition, including a significant amount of new material. All major kernel subsystems are included: the virtual memory (VM) system, processes and threads, the kernel dispatcher and scheduling classes, file systems and the virtual file system (VFS) framework, and core kernel facilities. New Solaris facilities for resource management are covered as well, along with a new chapter on networking. New features in Solaris 8 and Solaris 9 are called out as appropriate throughout the text. Examples of Solaris utilities and tools for performance and analysis work, described in the companion volume, are used throughout the text.

Solaris™ Performance and Tools. This book contains chapters on the tools and utilities bundled with Solaris 10: dtrace(1), mdb(1), kstat(1), etc. There are also extensive chapters on using the tools to analyze the performance andbehavior of a Solaris system.

The two texts are designed as companion volumes, and can be used in conjunction with access to the Solaris source code on

http://www.opensolaris.org

Readers interested in specific releases before Solaris 8 should continue to use the first edition as a reference.

Application developers can find information in these books about how Solaris OS implements functions behind the application programming interfaces. This information helps developers understand performance, scalability, and implementation specifics of each interface when they develop Solaris applications. The system overview section and sections on scheduling, interprocess communication, and file system behavior should be the most useful sections.

Device driver and kernel module developers of drivers, STREAMS modules, loadable system calls, etc., can find herein the general architecture and implementation theory of the Solaris OS. The Solaris kernel framework and facilities portions of the books (especially the locking and synchronization primitives chapters) are particularly relevant.

Systems administrators, systems analysts, database administrators, and Enterprise Resource Planning (ERP) managers responsible for performance tuning and capacity planning can learn about the behavioral characteristics of the major Solaris subsystems. The file system caching and memory management chapters provide a great deal of information about how Solaris behaves in real-world environments. The algorithms behind Solaris tunable parameters are covered in depth throughout the books.

Technical support staff responsible for the diagnosis, debugging, and support of Solaris will find a wealth of information about implementation details of Solaris. Major data structures and data flow diagrams are provided in each chapter to aid debugging and navigation of Solaris systems.

System users who just want to know more about how the Solaris kernel works will find high-level overviews at the start of each chapter.

Beyond the technical user community, those in academia studying operating systems will find that this text will work well as a reference. Solaris OS is a robust, feature-rich, volume production operating system, well suited to a variety of workloads, ranging from uniprocessor desktops to very large multiprocessor systems with large memory and input/output (I/O) configurations. The robustness and scalability of Solaris OS for commercial data processing, Web services, network applications, and scientific workloads is without peer in the industry. Much can be learned from studying such an operating system.

In June 2005, Sun Microsystems introduced OpenSolaris, a fully functional Solaris operating system release built from open source. As part of the OpenSolaris initiative, the Solaris kernel source was made generally available through an open license offering. This has some obvious benefits to this text. We can now include Solaris source directly in the text where appropriate, as well as refer to full source listings made available through the OpenSolaris initiative.

With OpenSolaris, a worldwide community of developers now has access to Solaris source code, and developers can contribute to whatever component of the operating system they find interesting. Source code accessibility allows us to structure the books such that we can cross-reference specific source files, right down to line numbers in the source tree.

OpenSolaris represents a significant milestone for technologists worldwide; a world-class, mature, robust, and feature-rich operating system is now easily accessible to anyone wishing to use Solaris, explore it, and contribute to its development.

Visit the Open Solaris Website to learn more about OpenSolaris:

http://www.opensolaris.org

The OpenSolaris source code is available at:

http://cvs.opensolaris.org/source

Source code references used throughout this text are relative to that starting location.

We organized the Solaris™ Internals volumes into several logical parts, each part grouping several chapters containing related information. Our goal was to provide a building block approach to the material by which later sections could build on information provided in earlier chapters. However, for readers familiar with particular aspects of operating systems design and implementation, the individual parts and chapters can stand on their own in terms of the subject matter they cover.

Part One: Introduction to Solaris Internals
Chapter 1 — Introduction

Part Two: The Process Model
Chapter 2 — The Solaris Process Model
Chapter 3 — Scheduling Classes and the Dispatcher
Chapter 4 — Interprocess Communication
Chapter 5 — Process Rights Management

Part Three: Resource Management
Chapter 6 — Zones
Chapter 7 — Projects, Tasks, and Resource Controls

Part Four: Memory
Chapter 8 — Introduction to Solaris Memory
Chapter 9 — Virtual Memory
Chapter 10 — Physical Memory
Chapter 11 — Kernel Memory
Chapter 12 — Hardware Address Translation
Chapter 13 — Working with Multiple Page Sizes in Solaris

Part Five: File Systems
Chapter 14 — File System Framework
Chapter 15 — The UFS File System

Part Six: Platform Specifics
Chapter 16 — Support for NUMA and CMT Hardware
Chapter 17 — Locking and Synchronization

Part Seven: Networking
Chapter 18 — The Solaris Network Stack

Part Eight: Kernel Services
Chapter 19 — Clocks and Timers
Chapter 20 — Task Queues
Chapter 21 — kmdb Implementation

Part One: Observability Methods
Chapter 1 — Introduction to Observability Tools
Chapter 2 — CPUs
Chapter 3 — Processes
Chapter 4 — Disk Behavior and Analysis
Chapter 5 — File Systems
Chapter 6 — Memory
Chapter 7 — Networks
Chapter 8 — Performance Counters
Chapter 9 — Kernel Monitoring

Part Two: Observability Infrastructure
Chapter 10 — Dynamic Tracing
Chapter 11 — Kernel Statistics

Part Three: Debugging
Chapter 12 — The Modular Debugger
Chapter 13 — An MDB Tutorial
Chapter 14 — Debugging Kernels

To complement these books, we created a Web site at which we will place updated material, tools we refer to, and links to related material on the topics covered. We will regularly update the Web site (http://www.solarisinternals.com) with information about this text and future work on Solaris™ Internals. The Web site will be enhanced to provide a forum for Frequently Asked Questions (FAQs) related to the text, as well as general questions about Solaris internals, performance, and behavior. If bugs are discovered in the text, we will post errata on the Web site as well.

Once again, a large investment in time and energy proved enormously rewarding for the authors. The support from Sun’s Solaris kernel development group, the Solaris user community, and readers of the first edition has been extremely gratifying. We believe we have been able to achieve more with the second edition in terms of providing Solaris users with a valuable reference text. We certainly extended our knowledge in writing it, and we look forward to hearing from readers.