From ahl@zday.sfbay.sun.com Wed Mar 31 10:28:40 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2VHSdkw019578 for ; Wed, 31 Mar 2010 10:28:39 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2VHSY3o006382; Wed, 31 Mar 2010 12:28:38 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L050020XPVNY600@nwk-avmta-2.sfbay.sun.com>; Wed, 31 Mar 2010 10:28:35 -0700 (PDT) Received: from zday.sfbay.sun.com ([10.6.46.178]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L050025GPVNTU00@nwk-avmta-2.sfbay.sun.com>; Wed, 31 Mar 2010 10:28:35 -0700 (PDT) Received: from zday.sfbay.sun.com (localhost [127.0.0.1]) by zday.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id o2VHans2156262; Wed, 31 Mar 2010 10:36:49 -0700 (PDT) Received: (from ahl@localhost) by zday.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id o2VHanuW156258; Wed, 31 Mar 2010 10:36:49 -0700 (PDT) Date: Wed, 31 Mar 2010 10:36:49 -0700 (PDT) From: Adam Leventhal Subject: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] To: PSARC-ext@sun.com Cc: Alan.Maguire@sun.com, ahl@eng.sun.com Message-id: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 9489 I'm submitting the following closed approved automatic fasttrack on behalf of Alan Maguire. The proposal has already been approved by the DTrace community. Binding is patch. Adam Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: DTrace TCP and UDP providers 1.2. Name of Document Author/Supplier: Author: Alan Maguire 1.3 Date of This Document: 31 March, 2010 4. Technical Description Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2010 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: DTrace TCP/UDP Providers 1.2. Name of Document Author/Supplier: Author: Brendan Gregg/Alan Maguire 1.3 Date of This Document: 08 March, 2010 4. Technical Description A. INTRODUCTION This case adds DTrace 'tcp' and 'udp' providers with probes for send and receive events. These providers cover the TCP and UDP protocol implementations in OpenSolaris respectively. In addition the tcp provider contains probes for TCP state machine transitions and significant events in connection processing (connection request, acceptance, refusal etc). The udp provider also contains probes which fire when a UDP socket is opened/closed. This is intended for use by customers for network observability and troubleshooting, and this work represents the second and third components of a suite of planned providers for the network stack. The first was described in PSARC/2008/302 DTrace IP Provider. B. DESCRIPTION This work will introduce the following probes for the 'tcp' provider: tcp:::send tcp:::receive tcp:::connect-request tcp:::connect-refused tcp:::connect-established tcp:::accept-established tcp:::accept-refused tcp:::state-change The arguments to these probes are: args[0] pktinfo_t * packet info args[1] csinfo_t * connection state info args[2] ipinfo_t * generic IP info args[3] tcpsinfo_t * TCP state information args[4] tcpinfo_t * TCP header args[5] tcpnsinfo_t * Next TCP state The order and content has been chosen for consistency with other network providers. The ipinfo_t * and tcpinfo_t * will be NULL for tcp:::state-change events. The csinfo_t * and tcpsinfo_t * will be NULL for tcp:::accept-refused events since there is no listening connection. The tcpnsinfo_t * argument is present for the tcp:::state-change probe only. The arguments contain: /* * pktinfo is where packet ID info can be made available for deeper * analysis if packet IDs become supported by the kernel in the future. * The pkt_addr member is currently always NULL. */ typedef struct pktinfo { uintptr_t pkt_addr; } pktinfo_t; /* * csinfo is where connection state info is made available. */ typedef struct csinfo { uintptr_t cs_addr; uint64_t cs_cid; pid_t cs_pid; zoneid_t cs_zoneid; } csinfo_t; Note that we have filled out the csinfo_t with additional information. In PSARC/2008/302 DTrace IP Provider, only the cs_addr was present. Here we take advantage of the fact that the IP datapath refactoring project modified IP such that a transmit attribute structure - ip_xmit_attr_t * - travels with the packet on the outbound path; and on the inbound path, the conn_t contains an ip_xmit_attr_t *, so that when a packet is classified by IP and passed up to UDP, SCTP or TCP, we have access to information about the target process, zone etc. We will also add a new connection id field to the ip_xmit_attr_t. Having such an identifier is extremely useful in DTrace. Future work will ensure this extended csinfo_t is used at the ip:::send probe points also, though significant code refactoring will be required to ensure availability of the ip_xmit_attr_t * at all these points. On the ip:::receive side, it is not possible to make this mapping as the IP packet has not yet been classified (and mapped to a conn_t) when ip:::receive fires. /* * ipinfo contains common IP info for both IPv4 and IPv6. */ typedef struct ipinfo { uint8_t ip_ver; /* IP version (4, 6) */ uint16_t ip_plength; /* payload length */ string ip_saddr; /* source address */ string ip_daddr; /* destination address */ } ipinfo_t; /* * tcpsinfo contains stable TCP details from tcp_t. */ typedef struct tcpsinfo { uintptr tcps_addr; int tcps_local; /* is delivered locally, boolean */ int tcps_active; /* active open (from here), boolean */ uint16_t tcps_lport; /* local port */ uint16_t tcps_fport; /* remote port */ string tcps_state; /* TCP state, as a string */ uint32_t tcps_iss; /* initial send sequence # */ uint32_t tcps_suna; /* sequence # sent but unacked */ uint32_t tcps_snxt; /* next sequence # to send */ uint32_t tcps_rack; /* sequence # we have acked */ uint32_t tcps_rnxt; /* next sequence # expected */ uint32_t tcps_swnd; /* send window size */ uint32_t tcps_snd_ws; /* send window scaling */ uint32_t tcps_rwnd; /* receive window size */ uint32_t tcps_rcv_ws; /* receive window scaling */ uint32_t tcps_rto; /* round-trip timeout, msec */ int tcps_retransmit; /* retransmit send event, boolean */ } tcpsinfo_t; /* * tcpinfo is the TCP header fields. */ typedef struct tcpinfo { uint16_t tcp_sport; /* source port */ uint16_t tcp_dport; /* destination port */ uint32_t tcp_seq; /* sequence number */ uint32_t tcp_ack; /* acknowledgment number */ uint8_t tcp_offset; /* data offset, in bytes */ uint8_t tcp_flags; /* flags */ uint16_t tcp_window; /* window size */ uint16_t tcp_checksum; /* checksum */ uint16_t tcp_urgent; /* urgent data pointer */ tcph_t *tcp_hdr; /* raw TCP header */ } tcpinfo_t; /* * tcpnsinfo provides the new tcp state for state changes. */ typedef struct tcpnsinfo { string tcps_state; /* TCP state, as a string */ } tcpnsinfo_t; This work will also introduce the following probes for the 'udp' provider: udp:::send udp:::receive udp:::socket-opened udp:::socket-closed The arguments to these probes are: args[0] pktinfo_t * packet info args[1] csinfo_t * connection state info args[2] ipinfo_t * generic IP info args[3] udpsinfo_t * UDP state information args[4] udpinfo_t * UDP header The ipinfo_t * and udpinfo_t * arguments are NULL for udp::socket-opened and udp:::socket-closed. /* * udpsinfo contains stable UDP details from udp_t. */ typedef struct udpsinfo { uintptr_t udps_addr; uint16_t upds_lport; /* local port */ uint16_t udps_fport; /* remote port */ } udpsinfo_t; /* * udpinfo is the UDP header fields. */ typedef struct udpinfo { uint16_t udp_sport; /* source port */ uint16_t udp_dport; /* destination port */ uint16_t udp_length; /* total length */ uint16_t udp_checksum; /* headers + data checksum */ udpha_t *udp_hdr; /* raw UDP header */ } udpinfo_t; C. EXAMPLES # Watch inbound TCP connections by remote address, dtrace -n 'tcp:::accept-established { trace(args[2]->ip_saddr); }' # Inbound TCP connections by destination port summary dtrace -n 'tcp:::accept-established { @port[args[3]->tcp_dport] = count(); }' # Watch inbound accepted TCP connections by process summary dtrace -n 'tcp:::accept-established { @cpid[args[1]->cs_pid] = count(); }' # Watch UDP total number of bytes sent/received by process dtrace -n 'udp:::send,udp:::receive { @bytes[args[1]->cs_pid] = sum(args[4]->udp_length);}' D. REFERENCES The suite of planned providers is described on the following website, which includes demonstrations and source from previous prototypes: http://www.opensolaris.org/os/community/dtrace/NetworkProvider These providers have also been discussed in the past on both dtrace-discuss and networking-discuss: http://www.opensolaris.org/jive/thread.jspa?messageID=57666 http://www.opensolaris.org/jive/thread.jspa?messageID=128518😆 E. DOCUMENTATION New chapters will be added to the current Solaris Dynamic Tracing Guide for these proposed providers, and demo scripts will be added to /usr/demo/dtrace. The tcp provider is described here: http://wikis.sun.com/display/DTrace/tcp+Provider ...and the udp provider is described here: http://wikis.sun.com/display/DTrace/udp+Provider F. STABILITY The DTrace internal stability table is described below: Element Name stability Data stability Dependency class Provider Evolving Evolving ISA Module Private Private Unknown Function Private Private Unknown Name Evolving Evolving ISA Arguments Evolving Evolving ISA 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: OS/Net 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: OS/Net 6.5. ARC review type: Automatic 6.6. ARC Exposure: open From darren.reed@oracle.com Wed Mar 31 22:33:04 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o315X4Lc001108 for ; Wed, 31 Mar 2010 22:33:04 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o315WnjB040013; Wed, 31 Mar 2010 23:33:03 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0600G0BNEDAL00@brm-avmta-1.central.sun.com>; Wed, 31 Mar 2010 23:32:37 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L06008YGNECHR50@brm-avmta-1.central.sun.com>; Wed, 31 Mar 2010 23:32:36 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o315WZpg009874; Thu, 01 Apr 2010 05:32:35 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2VG5bPf005559; Thu, 01 Apr 2010 05:32:31 +0000 (GMT) Received: from abhmt002.oracle.com by acsmt354.oracle.com with ESMTP id 129333801270099852; Wed, 31 Mar 2010 22:30:52 -0700 Received: from [129.145.154.61] (/129.145.154.61) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 31 Mar 2010 22:30:51 -0700 Date: Wed, 31 Mar 2010 22:30:50 -0700 From: Darren Reed Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> To: Adam Leventhal Cc: PSARC-ext@sun.com, alan.maguire@sun.com, ahl@eng.sun.com Message-id: <4BB42F8A.2030901@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BB42FF2.0106:SCFMA4539814,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 1839 I'm not sure that enough detail is provided for this to be "automatic" and there is also the problem of the missing UDP probes... see below. On 03/31/10 10:36, Adam Leventhal wrote: > B. DESCRIPTION > > This work will introduce the following probes for the 'tcp' provider: > > tcp:::send > tcp:::receive > tcp:::connect-request > tcp:::connect-refused > tcp:::connect-established > tcp:::accept-established > tcp:::accept-refused > tcp:::state-change > If we do simple trackng of accept-established and connect-established, we're able to count how many active connections there are but to properly adjust the counter when things close requires tracking state-change and understanding enough of TCP to know how to adjust your stats correctly. Futher to this, if I look at tcp:::connect-request and tcp:::state-change, is that enough or do I also need the other tcp:::connect's to detect when a connection request is refused? The state of the TCP socket should go from SYN-SENT to CLOSED for connect-refused. Similarly, connect-established should just be a alias for the transition of SYN-SENT to ESTABLISHED and accept-established a change from SYN-RECEIVED to ESTABLISHED. Is that all correct? Will a dtrace script that gathers connect-established and state-change have both of those probes activated for the same event? Given this, it would be appreciated if some text mentioned whether the above are actually aliases or not. With all of those state-change aliases available on the connect side, I'm curious about why the close side has been ignored. Is there a reason why we don't have "tcp:::closed"? Furthermore, whilst the case is titled "DTrace TCP and UDP providers", I only see TCP ones presented but an example with UDP. The materials need to be updated to specify which UDP ones are being introduced. Darren From garrett.damore@oracle.com Wed Mar 31 22:47:08 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o315l8UE001242 for ; Wed, 31 Mar 2010 22:47:08 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o315l2Xo051347; Wed, 31 Mar 2010 23:47:06 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0600305O2IEX00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 31 Mar 2010 22:47:06 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0600LCDO2HGI80@nwk-avmta-1.sfbay.Sun.COM>; Wed, 31 Mar 2010 22:47:05 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o315l3GF013769; Thu, 01 Apr 2010 05:47:03 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o311OdVH002467; Thu, 01 Apr 2010 05:47:02 +0000 (GMT) Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 129360131270100728; Wed, 31 Mar 2010 22:45:28 -0700 Received: from [10.7.251.172] (/10.7.251.172) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 31 Mar 2010 22:45:28 -0700 Date: Wed, 31 Mar 2010 22:45:25 -0700 From: "Garrett D'Amore" Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB42F8A.2030901@oracle.com> To: Darren Reed Cc: Adam Leventhal , PSARC-ext@sun.com, alan.maguire@sun.com, ahl@eng.sun.com Message-id: <4BB432F5.3020103@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4BB43357.00DA:SCFMA4539814,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 2081 Darren makes some good points. Please promote this to a fast track. - Garrett On 03/31/10 10:30 PM, Darren Reed wrote: > I'm not sure that enough detail is provided for > this to be "automatic" and there is also the problem > of the missing UDP probes... see below. > > On 03/31/10 10:36, Adam Leventhal wrote: >> B. DESCRIPTION >> >> This work will introduce the following probes for the 'tcp' provider: >> >> tcp:::send >> tcp:::receive >> tcp:::connect-request >> tcp:::connect-refused >> tcp:::connect-established >> tcp:::accept-established >> tcp:::accept-refused >> tcp:::state-change > > If we do simple trackng of accept-established and > connect-established, we're able to count how many > active connections there are but to properly adjust > the counter when things close requires tracking > state-change and understanding enough of TCP > to know how to adjust your stats correctly. > > Futher to this, if I look at tcp:::connect-request > and tcp:::state-change, is that enough or do I also > need the other tcp:::connect's to detect when a > connection request is refused? The state of the > TCP socket should go from SYN-SENT to > CLOSED for connect-refused. Similarly, > connect-established should just be a alias for > the transition of SYN-SENT to ESTABLISHED > and accept-established a change from > SYN-RECEIVED to ESTABLISHED. > Is that all correct? Will a dtrace script that > gathers connect-established and state-change > have both of those probes activated for the > same event? > > Given this, it would be appreciated if some text > mentioned whether the above are actually aliases > or not. > > With all of those state-change aliases available > on the connect side, I'm curious about why the > close side has been ignored. > > Is there a reason why we don't have "tcp:::closed"? > > Furthermore, whilst the case is titled "DTrace > TCP and UDP providers", I only see TCP ones > presented but an example with UDP. The materials > need to be updated to specify which UDP ones are > being introduced. > > Darren > From ahl@eng.sun.com Wed Mar 31 22:51:10 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o315pA8e001283 for ; Wed, 31 Mar 2010 22:51:10 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o315p6Gq018917; Thu, 1 Apr 2010 00:51:08 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L060040BO98NN00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 31 Mar 2010 22:51:08 -0700 (PDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0600LT5O97GG70@nwk-avmta-1.sfbay.Sun.COM>; Wed, 31 Mar 2010 22:51:07 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o315p1si010050; Thu, 01 Apr 2010 05:51:02 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2VKclYX010485; Thu, 01 Apr 2010 05:51:01 +0000 (GMT) Received: from abhmt005.oracle.com by acsmt355.oracle.com with ESMTP id 129368621270100988; Wed, 31 Mar 2010 22:49:48 -0700 Received: from [192.168.1.101] (/24.130.173.33) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 31 Mar 2010 22:49:47 -0700 Date: Wed, 31 Mar 2010 22:49:46 -0700 From: Adam Leventhal Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB42F8A.2030901@oracle.com> To: Darren Reed Cc: PSARC-ext@sun.com, Alan Maguire , Adam Leventhal Message-id: <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1077) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BB43445.0162,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> Status: RO Content-Length: 2020 Darren, > I'm not sure that enough detail is provided for > this to be "automatic" and there is also the problem > of the missing UDP probes... see below. If you scroll down, you'll find the udp provider probes listed. > If we do simple trackng of accept-established and > connect-established, we're able to count how many > active connections there are but to properly adjust > the counter when things close requires tracking > state-change and understanding enough of TCP > to know how to adjust your stats correctly. > > Futher to this, if I look at tcp:::connect-request > and tcp:::state-change, is that enough or do I also > need the other tcp:::connect's to detect when a > connection request is refused? The state of the > TCP socket should go from SYN-SENT to > CLOSED for connect-refused. Similarly, > connect-established should just be a alias for > the transition of SYN-SENT to ESTABLISHED > and accept-established a change from > SYN-RECEIVED to ESTABLISHED. > Is that all correct? Will a dtrace script that > gathers connect-established and state-change > have both of those probes activated for the > same event? > > Given this, it would be appreciated if some text > mentioned whether the above are actually aliases > or not. > > With all of those state-change aliases available > on the connect side, I'm curious about why the > close side has been ignored. It sounds like you're raising an issue of documentation and you have a question about how a specific query would be phrased. Is that accurate? > Is there a reason why we don't have "tcp:::closed"? I'm sure Alan can address that. > Furthermore, whilst the case is titled "DTrace > TCP and UDP providers", I only see TCP ones > presented but an example with UDP. The materials > need to be updated to specify which UDP ones are > being introduced. Again, I think a thorough reading of the case will reveal that that information was indeed included. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From Alan.Maguire@sun.com Thu Apr 1 10:31:30 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31HVTjV001162 for ; Thu, 1 Apr 2010 10:31:30 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31HUsMZ016111 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 1 Apr 2010 12:31:29 -0500 (CDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700J2JKNVSI00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 01 Apr 2010 11:31:07 -0600 (MDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L07004TWKNSGKB0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 01 Apr 2010 11:31:05 -0600 (MDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o31HV4Kb011279 for ; Thu, 01 Apr 2010 17:31:04 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0700E00KGMJF00@fe-emea-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 01 Apr 2010 18:30:41 +0100 (BST) Received: from [192.168.1.8] ([unknown] [86.44.91.240]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L07005Z4KN3XAA0@fe-emea-10.sun.com>; Thu, 01 Apr 2010 18:30:41 +0100 (BST) Date: Thu, 01 Apr 2010 18:30:39 +0100 From: Alan Maguire Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB44C30.9040105@oracle.com> Sender: Alan.Maguire@sun.com To: Darren Reed Cc: Adam Leventhal , PSARC-ext@sun.com Message-id: <4BB4D83F.201@sun.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_TXworZbP8ltve8YnhHd4/Q)" X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 16149 This is a multi-part message in MIME format. --Boundary_(ID_TXworZbP8ltve8YnhHd4/Q) Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT On 01/04/2010 08:33, Darren Reed wrote: > On 03/31/10 22:49, Adam Leventhal wrote: >> Darren, >> >> >>> I'm not sure that enough detail is provided for >>> this to be "automatic" and there is also the problem >>> of the missing UDP probes... see below. >>> >> >> If you scroll down, you'll find the udp provider probes listed. >> >> >>> If we do simple trackng of accept-established and >>> connect-established, we're able to count how many >>> active connections there are but to properly adjust >>> the counter when things close requires tracking >>> state-change and understanding enough of TCP >>> to know how to adjust your stats correctly. >>> True. With respect to tcp close issues, I guess there's a few ways to handle close events: 1. a socket-oriented close probe "tcp:::socket-close" (and for consistency tcp:::socket-open). Adding these seems reasonable to me, especially given the fact that it would be consistent with the udp provider, as you note below. 2. a close probe in the style of the connect-* and accept-* probes - "tcp:::connection-closed" perhaps? This could fire when tcp reaches the fictional closed state for one of the following reasons: - when the TIME_WAIT timeout on a closing connection has expired - when we receive the LAST_ACK for a passive close - when we time out in SYN_SENT state - when we send or receive a RST segment aborting the connection, or we receive a RST|ACK. 3. a state-change probe for the closed state (tcp:::state-change where the new state argument is "state-closed"). This was discussed a while back, and the consensus was not to use the closed state since it is described as fictional in RFC793. Jim describes this here: http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html So my proposal would be to add 1 and 2 above to the provider specification. Does this seem reasonable? >>> Futher to this, if I look at tcp:::connect-request >>> and tcp:::state-change, is that enough or do I also >>> need the other tcp:::connect's to detect when a >>> connection request is refused? The state of the >>> TCP socket should go from SYN-SENT to >>> CLOSED for connect-refused. Similarly, >>> connect-established should just be a alias for >>> the transition of SYN-SENT to ESTABLISHED >>> and accept-established a change from >>> SYN-RECEIVED to ESTABLISHED. >>> I'll flesh out the details in the provider specification a bit more. Is the following is what you're looking for? tcp:::connect-request Fires when the initial SYN is sent as a result of a connect(3SOCKET). The state then changes to SYN_SENT. tcp:::connect-refused Fires when a RST|ACK connection refusal is received in response to a SYN segment. tcp:::connect-established Fires when the final ACK in the three-way handshake has been sent - the state then changes from SYN_SENT to ESTABLISHED. tcp:::accept-established Fires when the final ack in the three-way handshake has been received - the state then changes from SYN_RCVD to ESTABLISHED. tcp:::accept-refused Fires when sending a RST|ACK connection refusal. >>> Is that all correct? Will a dtrace script that >>> gathers connect-established and state-change >>> have both of those probes activated for the >>> same event? >>> >>> Yes. >>> Given this, it would be appreciated if some text >>> mentioned whether the above are actually aliases >>> or not. >>> >>> Sure. The rationale isn't quite that they are aliases - the connect-* and accept-* probes are oriented towards representing not just the internal TCP state as it changes, but also the TCP segments we send or receive that drive those changes. More concretely, for the tcp:::state-change probes, the ipinfo_t * and tcpinfo_t * (representing IP and TCP headers associated with the probe event) are NULL, whereas for the tcp:::connect-* and tcp:::accept-* probes they point at the IP and TCP headers associated with the event. >>> With all of those state-change aliases available >>> on the connect side, I'm curious about why the >>> close side has been ignored. >>> >> >> It sounds like you're raising an issue of documentation and you have a question about how a specific query would be phrased. Is that accurate? >> > > Yes. More information is required about how the state-change > probe relates to all of the other TCP probes that involve state > changes of the socket. > Is the above description sufficient? If so, I'll update the provider specification accordingly. >>> Is there a reason why we don't have "tcp:::closed"? >>> >> >> I'm sure Alan can address that. >> > > I suppose that answer should also address where there is no > tcp:::socket-open and tcp:::socket-closed. > > See above. >>> Furthermore, whilst the case is titled "DTrace >>> TCP and UDP providers", I only see TCP ones >>> presented but an example with UDP. The materials >>> need to be updated to specify which UDP ones are >>> being introduced. >>> >> >> Again, I think a thorough reading of the case will reveal that that information was indeed included. >> > > Ah, they just weren't where I expected them to be. > Apologies. > > Was there any consideration given to supporting both > "udp:::connect-request" and "udp:::connect-refused"? > Hmm, it's a good idea but I'm not sure if we should have connect-request as a probe point in UDP. It would definitely be useful to have a probe point for when we issue or receive ICMP port unreachable errors - perhaps we should stay away from the "connect-" terminology though, since UDP is a connectionless protocol. Maybe udp:::port-unreachable-send udp:::port-unreachable-receive ? > It seems that the architecture being presented is that the UDP > probes fit around UDP sockets (socket-open & socket-closed) > whereas TCP probes fit around something else - presumably > TCP connections? That was true, but I think you've raised some good points that move us to a more consistent architecture across providers. > > This case updates csinfo_t but makes no mention of what > the role of cs_addr is... is it safe to assume it is still NULL? > Apologies - I should have specified that cs_addr is a pointer to the ip_xmit_attr_t which we translated into a csinfo_t. > No comments are provided about udps_addr or tcps_addr. > Is their behaviour the same as cs_addr - NULL? > These will be pointers to the udp_t and tcp_t used to translate into the udpinfo_t/tcpinfo_t. Thanks! Alan --Boundary_(ID_TXworZbP8ltve8YnhHd4/Q) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: 7BIT On 01/04/2010 08:33, Darren Reed wrote:
On 03/31/10 22:49, Adam Leventhal wrote: 
Darren,

  
I'm not sure that enough detail is provided for
this to be "automatic" and there is also the problem
of the missing UDP probes... see below.
    

If you scroll down, you'll find the udp provider probes listed.

  
If we do simple trackng of accept-established and
connect-established, we're able to count how many
active connections there are but to properly adjust
the counter when things close requires tracking
state-change and understanding enough of TCP
to know how to adjust your stats correctly.
True. With respect to tcp close issues, I guess there's
a few ways to handle close events:

1. a socket-oriented close probe "tcp:::socket-close" (and
for consistency tcp:::socket-open). Adding these seems
reasonable to me, especially given the fact that it would
be consistent with the udp provider, as you note below.

2. a close probe in the style of the connect-* and
accept-* probes - "tcp:::connection-closed" perhaps?
This could fire when tcp reaches the fictional
closed state for one of the following reasons:

- when the TIME_WAIT timeout on a closing connection
has expired
- when we receive the LAST_ACK for a passive close
- when we time out in SYN_SENT state
- when we send or receive a RST segment aborting
the connection, or we receive a RST|ACK.

3. a state-change probe for the closed state
(tcp:::state-change where the new state argument is
"state-closed"). This  was discussed a while back, and the
consensus was not to use the closed state since it is described
as fictional in RFC793. Jim describes this here:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html

So my proposal would be to add 1 and 2 above to the
provider specification. Does this seem reasonable?

Futher to this, if I look at tcp:::connect-request
and tcp:::state-change, is that enough or do I also
need  the other tcp:::connect's to detect when a
connection request is refused? The state of the
TCP socket should go from SYN-SENT to
CLOSED for connect-refused. Similarly,
connect-established should just be a alias for
the transition of SYN-SENT to ESTABLISHED
and accept-established a change from
SYN-RECEIVED to ESTABLISHED.
I'll flesh out the details in the provider specification
a bit more. Is the following is what you're looking
for?

tcp:::connect-request

Fires when the initial SYN is sent as a result of
a connect(3SOCKET). The state then changes to SYN_SENT.

tcp:::connect-refused

Fires when a RST|ACK connection refusal is received
in response to a SYN segment.
 
tcp:::connect-established

Fires when the final ACK in the three-way handshake has
been sent - the state then changes from SYN_SENT to ESTABLISHED.

tcp:::accept-established

Fires when the final ack in the three-way handshake has
been received - the state then changes from SYN_RCVD to ESTABLISHED.

tcp:::accept-refused

Fires when sending a RST|ACK connection refusal.


Is that all correct? Will a dtrace script that
gathers connect-established and state-change
have both of those probes activated for the
same event?
Yes.
Given this, it would be appreciated if some text
mentioned whether the above are actually aliases
or not.
Sure. The rationale isn't quite that they are
aliases - the connect-* and accept-* probes
are oriented towards representing not just
the internal TCP state as it changes, but also
the TCP segments we send or receive that drive
those changes. More concretely, for the tcp:::state-change
probes, the ipinfo_t * and tcpinfo_t * (representing
IP and TCP headers associated with the probe
event) are NULL, whereas for the tcp:::connect-*
and tcp:::accept-* probes they point at the IP and
TCP headers associated with the event. 
With all of those state-change aliases available
on the connect side, I'm curious about why the
close side has been ignored.
    

It sounds like you're raising an issue of documentation and you have a question about how a specific query would be phrased. Is that accurate?

Yes. More information is required about how the state-change
probe relates to all of the other TCP probes that involve state
changes of the socket.

Is the above description sufficient? If so, I'll update
the provider specification accordingly.
Is there a reason why we don't have "tcp:::closed"?
    

I'm sure Alan can address that.

I suppose that answer should also address where there is no
tcp:::socket-open and tcp:::socket-closed.


See above.
Furthermore, whilst the case is titled "DTrace
TCP and UDP providers", I only see TCP ones
presented but an example with UDP. The materials
need to be updated to specify which UDP ones are
being introduced.
    

Again, I think a thorough reading of the case will reveal that that information was indeed included.

Ah, they just weren't where I expected them to be.
Apologies.

Was there any consideration given to supporting both
"udp:::connect-request" and "udp:::connect-refused"?

Hmm, it's a good idea but I'm not sure if we should have
connect-request as a probe point in UDP. It would definitely
be useful to have a probe point for when we issue or receive
ICMP port unreachable errors - perhaps we should stay
away from the "connect-" terminology though, since UDP
is a connectionless protocol. Maybe

udp:::port-unreachable-send
udp:::port-unreachable-receive

?

It seems that the architecture being presented is that the UDP
probes fit around UDP sockets (socket-open & socket-closed)
whereas TCP probes fit around something else - presumably
TCP connections?
That was true, but I think you've raised some good points
that move us to a more consistent architecture across
providers.

This case updates csinfo_t but makes no mention of what
the role of cs_addr is... is it safe to assume it is still NULL?

Apologies - I should have specified that cs_addr is a pointer
to the ip_xmit_attr_t which we translated into a csinfo_t.
No comments are provided about udps_addr or tcps_addr.
Is their behaviour the same as cs_addr - NULL?

These will be pointers to the udp_t and tcp_t used to translate
into the udpinfo_t/tcpinfo_t.

Thanks!

Alan
--Boundary_(ID_TXworZbP8ltve8YnhHd4/Q)-- From Alan.Maguire@Sun.COM Thu Apr 1 12:54:21 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31JsLDZ003595 for ; Thu, 1 Apr 2010 12:54:21 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31JsLiX018006 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 1 Apr 2010 12:54:21 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700903RALHO00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@SUN.COM); Thu, 01 Apr 2010 12:54:21 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L07002ZFRAJ5490@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@SUN.COM); Thu, 01 Apr 2010 12:54:20 -0700 (PDT) Received: from fe-emea-13.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o31JsJcv012994 for ; Thu, 01 Apr 2010 19:54:19 +0000 (GMT) Received: from conversion-daemon.fe-emea-13.sun.com by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0700900R5E8100@fe-emea-13.sun.com> for PSARC-ext@SUN.COM (ORCPT PSARC-ext@SUN.COM); Thu, 01 Apr 2010 20:53:57 +0100 (BST) Received: from [192.168.1.8] ([unknown] [86.44.91.240]) by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L0700895R9XY360@fe-emea-13.sun.com>; Thu, 01 Apr 2010 20:53:57 +0100 (BST) Date: Thu, 01 Apr 2010 20:53:56 +0100 From: Alan Maguire Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB4D83F.201@sun.com> Sender: Alan.Maguire@Sun.COM To: Darren Reed Cc: Adam Leventhal , PSARC-ext@Sun.COM Message-id: <4BB4F9D4.4010207@sun.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_8Vc/ualUBtQMPzN036fjIg)" X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 5880 This is a multi-part message in MIME format. --Boundary_(ID_8Vc/ualUBtQMPzN036fjIg) Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT Apologies, I misstated something below.... On 01/04/2010 18:30, Alan Maguire wrote: > On 01/04/2010 08:33, Darren Reed wrote: >> On 03/31/10 22:49, Adam Leventhal wrote: >>> Darren, >>> >>> >>>> I'm not sure that enough detail is provided for >>>> this to be "automatic" and there is also the problem >>>> of the missing UDP probes... see below. >>>> >>> >>> If you scroll down, you'll find the udp provider probes listed. >>> >>> >>>> If we do simple trackng of accept-established and >>>> connect-established, we're able to count how many >>>> active connections there are but to properly adjust >>>> the counter when things close requires tracking >>>> state-change and understanding enough of TCP >>>> to know how to adjust your stats correctly. >>>> > True. With respect to tcp close issues, I guess there's > a few ways to handle close events: > > 1. a socket-oriented close probe "tcp:::socket-close" (and > for consistency tcp:::socket-open). Adding these seems > reasonable to me, especially given the fact that it would > be consistent with the udp provider, as you note below. > > 2. a close probe in the style of the connect-* and > accept-* probes - "tcp:::connection-closed" perhaps? > This could fire when tcp reaches the fictional > closed state for one of the following reasons: > > - when the TIME_WAIT timeout on a closing connection > has expired > - when we receive the LAST_ACK for a passive close > - when we time out in SYN_SENT state > - when we send or receive a RST segment aborting > the connection, or we receive a RST|ACK. > > 3. a state-change probe for the closed state > (tcp:::state-change where the new state argument is > "state-closed"). This was discussed a while back, and the > consensus was not to use the closed state since it is described > as fictional in RFC793. Jim describes this here: > > http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html > > So my proposal would be to add 1 and 2 above to the > provider specification. Does this seem reasonable? > Actually, we already have 3 - tcp:::state-change where the new state is "state-closed". Apologies for the confusion - I was writing from memory. So the question is if we should add 1 and 2 I guess. Alan --Boundary_(ID_8Vc/ualUBtQMPzN036fjIg) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Apologies, I misstated something below....

On 01/04/2010 18:30, Alan Maguire wrote:
On 01/04/2010 08:33, Darren Reed wrote:
On 03/31/10 22:49, Adam Leventhal wrote: 
Darren,

  
I'm not sure that enough detail is provided for
this to be "automatic" and there is also the problem
of the missing UDP probes... see below.
    

If you scroll down, you'll find the udp provider probes listed.

  
If we do simple trackng of accept-established and
connect-established, we're able to count how many
active connections there are but to properly adjust
the counter when things close requires tracking
state-change and understanding enough of TCP
to know how to adjust your stats correctly.
True. With respect to tcp close issues, I guess there's
a few ways to handle close events:

1. a socket-oriented close probe "tcp:::socket-close" (and
for consistency tcp:::socket-open). Adding these seems
reasonable to me, especially given the fact that it would
be consistent with the udp provider, as you note below.

2. a close probe in the style of the connect-* and
accept-* probes - "tcp:::connection-closed" perhaps?
This could fire when tcp reaches the fictional
closed state for one of the following reasons:

- when the TIME_WAIT timeout on a closing connection
has expired
- when we receive the LAST_ACK for a passive close
- when we time out in SYN_SENT state
- when we send or receive a RST segment aborting
the connection, or we receive a RST|ACK.

3. a state-change probe for the closed state
(tcp:::state-change where the new state argument is
"state-closed"). This  was discussed a while back, and the
consensus was not to use the closed state since it is described
as fictional in RFC793. Jim describes this here:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html

So my proposal would be to add 1 and 2 above to the
provider specification. Does this seem reasonable?

Actually, we already have 3 - tcp:::state-change where
the new state is "state-closed". Apologies for the confusion -
I was writing from memory. So the question is if we should
add 1 and 2 I guess.

Alan
--Boundary_(ID_8Vc/ualUBtQMPzN036fjIg)-- From Alan.Maguire@sun.com Fri Apr 2 03:53:41 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32ArfjX002926 for ; Fri, 2 Apr 2010 03:53:41 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32Are3p019738 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 2 Apr 2010 05:53:41 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800B0HWXH3900@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@SUN.COM); Fri, 02 Apr 2010 03:53:41 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800LJDWXFM910@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@SUN.COM); Fri, 02 Apr 2010 03:53:40 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o32Ardts011372 for ; Fri, 02 Apr 2010 10:53:39 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0800C00WQDSQ00@fe-emea-10.sun.com> for PSARC-ext@SUN.COM (ORCPT PSARC-ext@SUN.COM); Fri, 02 Apr 2010 11:53:33 +0100 (BST) Received: from [192.168.1.8] ([unknown] [86.44.91.240]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L0800967WX8TV90@fe-emea-10.sun.com>; Fri, 02 Apr 2010 11:53:33 +0100 (BST) Date: Fri, 02 Apr 2010 11:53:33 +0100 From: Alan Maguire Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB4F9D4.4010207@sun.com> Sender: Alan.Maguire@sun.com To: Darren Reed Cc: Adam Leventhal , PSARC-ext@sun.com Message-id: <4BB5CCAD.9090503@sun.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_lgyM1UqOndh2fiehVpEBHw)" X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 12410 This is a multi-part message in MIME format. --Boundary_(ID_lgyM1UqOndh2fiehVpEBHw) Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT Sorry to be following up again, but I think it would be helpful if I make a specific proposal wrt the additional probes required, especially the tcp close-related probes. I think adding the following set of probes to the original proposal should hopefully address shortcomings in this area in a manner consistent with the connect-* and accept-* probes: tcp:::close-request Fires when a FIN segment is sent to a peer, either as initiation of a graceful shutdown or in response to a shutdown request from a peer. tcp:::close-established Fires when an ACK is received to a previously-sent FIN segment. This probe firing may simply indicate a half-close - i.e. the peer connection may not close from it's side. tcp:::close-accepted Fires when a FIN segment is received from a peer. If tcp:::close-accepted and tcp:::close-established fire, the connection has shut down both sides but may linger of course. tcp:::reset-request Fires when a RST segment is sent to a peer connection to forcibly close (e.g. when we close and SO_LINGER is set to 0). This is distinguished from a RST|ACK segment, the sending of which is covered by the tcp:::accept-refused probe. tcp:::reset-accepted Fires when a RST segment is received from a peer connection. With these probes in hand, we can ask questions like: - how long do connections take to close down? - which side initiated the close? - how many half-closed connections are there? - how many connections close gracefully? how many close forcefully? etc. The naming of the probes is consistent with the connect-* and accept-* probes, and like those probes they supply the IP/TCP headers associated with the control segment they represent. The addition of these probes also means we have coverage for all the connection control segments (SYN, FIN, RST) and their related acknowledgments. We should also add: tcp:::socket-opened Fires when a TCP socket is opened. tcp:::socket-closed Fires when a TCP socket is closed. In addition to these, we also have the tcp:::state-change probe which covers the "closed" pseudo-state. For UDP, as Darren suggests, it would be helpful to have equivalents for the accept-refused/connect-refused probes so that users can detect cases where they attempt to connect to nonexistent services. I think it may make sense to stay away from connect-refused/accept-refused for names as these terms imply a connection, so I suggest (the rather unwieldy): udp:::port-unreachable-send Fires when an ICMP port unreachable message is sent in response to a UDP datagram heading for a port with no UDP server. It is true that an ICMP DTrace provider should cover this event, but it might be helpful to have a probe point within the UDP provider for such an event also. udp:::port-unreachable-receive Fires when an ICMP port unreachable message is received in response to a UDP datagram. Thanks! Alan On 01/04/2010 20:53, Alan Maguire wrote: > Apologies, I misstated something below.... > > On 01/04/2010 18:30, Alan Maguire wrote: >> On 01/04/2010 08:33, Darren Reed wrote: >>> On 03/31/10 22:49, Adam Leventhal wrote: >>>> Darren, >>>> >>>> >>>>> I'm not sure that enough detail is provided for >>>>> this to be "automatic" and there is also the problem >>>>> of the missing UDP probes... see below. >>>>> >>>> >>>> If you scroll down, you'll find the udp provider probes listed. >>>> >>>> >>>>> If we do simple trackng of accept-established and >>>>> connect-established, we're able to count how many >>>>> active connections there are but to properly adjust >>>>> the counter when things close requires tracking >>>>> state-change and understanding enough of TCP >>>>> to know how to adjust your stats correctly. >>>>> >> True. With respect to tcp close issues, I guess there's >> a few ways to handle close events: >> >> 1. a socket-oriented close probe "tcp:::socket-close" (and >> for consistency tcp:::socket-open). Adding these seems >> reasonable to me, especially given the fact that it would >> be consistent with the udp provider, as you note below. >> >> 2. a close probe in the style of the connect-* and >> accept-* probes - "tcp:::connection-closed" perhaps? >> This could fire when tcp reaches the fictional >> closed state for one of the following reasons: >> >> - when the TIME_WAIT timeout on a closing connection >> has expired >> - when we receive the LAST_ACK for a passive close >> - when we time out in SYN_SENT state >> - when we send or receive a RST segment aborting >> the connection, or we receive a RST|ACK. >> >> 3. a state-change probe for the closed state >> (tcp:::state-change where the new state argument is >> "state-closed"). This was discussed a while back, and the >> consensus was not to use the closed state since it is described >> as fictional in RFC793. Jim describes this here: >> >> http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html >> >> So my proposal would be to add 1 and 2 above to the >> provider specification. Does this seem reasonable? >> > Actually, we already have 3 - tcp:::state-change where > the new state is "state-closed". Apologies for the confusion - > I was writing from memory. So the question is if we should > add 1 and 2 I guess. > > Alan --Boundary_(ID_lgyM1UqOndh2fiehVpEBHw) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Sorry to be following up again, but I think it would
be helpful if I make a specific proposal wrt the
additional probes required, especially the tcp
close-related probes. I think adding the following
set of probes to the original proposal should hopefully
address shortcomings in this area in a manner
consistent with the connect-* and accept-* probes:

tcp:::close-request

Fires when a FIN segment is sent to a peer, either
as initiation of a graceful shutdown or in response
to a shutdown request from a peer.

tcp:::close-established

Fires when an ACK is received to a previously-sent
FIN segment. This probe firing may simply indicate
a half-close - i.e. the peer connection may not close
from it's side.

tcp:::close-accepted

Fires when a FIN segment is received from a peer.
If tcp:::close-accepted and tcp:::close-established
fire, the connection has shut down both sides but
may linger of course.

tcp:::reset-request

Fires when a RST segment is sent to a peer connection
to forcibly close (e.g. when we close and SO_LINGER is
set to 0). This is distinguished from a RST|ACK segment,
the sending of which is covered by the tcp:::accept-refused
probe.

tcp:::reset-accepted

Fires when a RST segment is received from a peer
connection.

With these probes in hand, we can ask questions like:

- how long do connections take to close down?
- which side initiated the close?
- how many half-closed connections are there?
- how many connections close gracefully? how many
close forcefully?

etc.

The naming of the probes is consistent with the
connect-* and accept-* probes, and like those
probes they supply the IP/TCP headers associated
with the control segment they represent. The
addition of these probes also means we have
coverage for all the connection control segments
(SYN, FIN, RST) and their related acknowledgments.

We should also add:

tcp:::socket-opened

Fires when a TCP socket is opened.

tcp:::socket-closed

Fires when a TCP socket is closed.

In addition to these, we also have the
tcp:::state-change probe which covers the
"closed" pseudo-state.

For UDP, as Darren suggests, it would be helpful
to have equivalents for the
accept-refused/connect-refused probes so that
users can detect cases where they attempt
to connect to nonexistent services. I think it
may make sense to stay away from
connect-refused/accept-refused for names as
these terms imply a connection, so I suggest
(the rather unwieldy):

udp:::port-unreachable-send

Fires when an ICMP port unreachable message
is sent in response to a UDP datagram heading
for a port with no UDP server. It is true that
an ICMP DTrace provider should cover this event,
but it might be helpful to have a probe point
within the UDP provider for such an event also.

udp:::port-unreachable-receive

Fires when an ICMP port unreachable message
is received in response to a UDP datagram.

Thanks!

Alan

On 01/04/2010 20:53, Alan Maguire wrote:
Apologies, I misstated something below....

On 01/04/2010 18:30, Alan Maguire wrote:
On 01/04/2010 08:33, Darren Reed wrote:
On 03/31/10 22:49, Adam Leventhal wrote: 
Darren,

  
I'm not sure that enough detail is provided for
this to be "automatic" and there is also the problem
of the missing UDP probes... see below.
    

If you scroll down, you'll find the udp provider probes listed.

  
If we do simple trackng of accept-established and
connect-established, we're able to count how many
active connections there are but to properly adjust
the counter when things close requires tracking
state-change and understanding enough of TCP
to know how to adjust your stats correctly.
True. With respect to tcp close issues, I guess there's
a few ways to handle close events:

1. a socket-oriented close probe "tcp:::socket-close" (and
for consistency tcp:::socket-open). Adding these seems
reasonable to me, especially given the fact that it would
be consistent with the udp provider, as you note below.

2. a close probe in the style of the connect-* and
accept-* probes - "tcp:::connection-closed" perhaps?
This could fire when tcp reaches the fictional
closed state for one of the following reasons:

- when the TIME_WAIT timeout on a closing connection
has expired
- when we receive the LAST_ACK for a passive close
- when we time out in SYN_SENT state
- when we send or receive a RST segment aborting
the connection, or we receive a RST|ACK.

3. a state-change probe for the closed state
(tcp:::state-change where the new state argument is
"state-closed"). This  was discussed a while back, and the
consensus was not to use the closed state since it is described
as fictional in RFC793. Jim describes this here:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html

So my proposal would be to add 1 and 2 above to the
provider specification. Does this seem reasonable?

Actually, we already have 3 - tcp:::state-change where
the new state is "state-closed". Apologies for the confusion -
I was writing from memory. So the question is if we should
add 1 and 2 I guess.

Alan

--Boundary_(ID_lgyM1UqOndh2fiehVpEBHw)-- From sebastien.roy@oracle.com Fri Apr 2 05:53:16 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32CrGuN004230 for ; Fri, 2 Apr 2010 05:53:16 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32CrEQw011563; Fri, 2 Apr 2010 05:53:15 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900A1B2GQME00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 06:53:14 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L09008WA2GQLQ00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 06:53:14 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o32CrDCV024089; Fri, 02 Apr 2010 12:53:13 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31HEQTb013086; Fri, 02 Apr 2010 12:53:12 +0000 (GMT) Received: from abhmt007.oracle.com by acsmt354.oracle.com with ESMTP id 133065851270212700; Fri, 02 Apr 2010 05:51:40 -0700 Received: from [10.7.250.40] (/10.7.250.40) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 05:51:40 -0700 Date: Fri, 02 Apr 2010 08:51:38 -0400 From: Sebastien Roy Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB5CCAD.9090503@sun.com> To: Alan Maguire Cc: Darren Reed , PSARC-ext@sun.com, Adam Leventhal Message-id: <4BB5E85A.8010502@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BB5E8B9.000D:SCFMA4539814,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100302 Lightning/1.0b1 Thunderbird/3.0.2 Status: RO Content-Length: 616 Alan and Adam, On 04/ 2/10 06:53 AM, Alan Maguire wrote: > Sorry to be following up again, but I think it would > be helpful if I make a specific proposal wrt the > additional probes required, especially the tcp > close-related probes. I think adding the following > set of probes to the original proposal should hopefully > address shortcomings in this area in a manner > consistent with the connect-* and accept-* probes: Given the discussion this case has generated and the fact that the materials are undergoing change that needs to be reviewed, I would suggest upgrading this to a fast-track. Thanks, -Seb From Nicolas.Williams@sun.com Fri Apr 2 09:48:21 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32GmLcF008482 for ; Fri, 2 Apr 2010 09:48:21 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32GmIr2001054; Fri, 2 Apr 2010 11:48:19 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900L0RDCI1E00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 09:48:18 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L09008E9DCIIW20@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 09:48:18 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o32GmHjc013216; Fri, 02 Apr 2010 16:48:17 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o32FsGrv016268; Fri, 02 Apr 2010 16:48:16 +0000 (GMT) Received: from abhmt003.oracle.com by acsmt353.oracle.com with ESMTP id 133539941270226894; Fri, 02 Apr 2010 09:48:14 -0700 Received: from Sun.COM (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 09:48:14 -0700 Date: Fri, 02 Apr 2010 11:48:09 -0500 From: Nicolas Williams Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB5CCAD.9090503@sun.com> To: Alan Maguire Cc: Darren Reed , Adam Leventhal , PSARC-ext@sun.com Message-id: <20100402164809.GW4225@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4BB61FD0.018E,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 2545 On Fri, Apr 02, 2010 at 11:53:33AM +0100, Alan Maguire wrote: > [...]. I think adding the following > set of probes to the original proposal should hopefully > address shortcomings in this area in a manner > consistent with the connect-* and accept-* probes: > > tcp:::close-request > > Fires when a FIN segment is sent to a peer, either > [...] > > tcp:::close-established > > Fires when an ACK is received to a previously-sent > FIN segment. This probe firing may simply indicate > [...] > > tcp:::close-accepted > > Fires when a FIN segment is received from a peer. > [...] > > tcp:::reset-request > > Fires when a RST segment is sent to a peer connection > [...] > > tcp:::reset-accepted > > [...] I'd rather see probes named and specified unambiguously after: - TCP state transitions (where the name of the probe that fires represents the new state of a connection and the old state is in some argument). State names should be taken from the standard: CLOSED, LISTEN, SYN_RCVD, SYN_SENT, EXTABLISHED, CLOSE_WAIT, LAST_ACK, FIN_WAIT_1, CLOSING, FIN_WAIT_2, TIME_WAIT. You might have just one probe, with the old and new states being arguments. - TCP "signals": locally-initiated close/shutdown, active open (connect), passive open (listen), receipt of RST, SYN, SYN-ACK, FIN, FIN-ACK, and ACK of a SYN-ACK, FIN, or FIN-ACK. These should also be named after the terminology used in the standard. Whether a signal is "accepted" or "rejected" (either dropped silently or responded to via a TCP or ICMP message) should be an argument of the TCP signals probes. You might have just one probe, with the signal being an argument. (Or two probes, one for locally generated signals and one for remote signals.) - TCP options? A probe per-option? Or let the caller parse options data passed as arguments to the relevant probes? TCP options can have a huge impact on perfomance (think of SACK, window scaling, PAWS, ...). Arguably the options that apply to a given connection should always be available for any probes firing for any connection in at least ESTABLISHED state. - data send/receive, ACKs of data - window size changes - congestion state transitions - PMTUD events The last five are probably crucial for performance analysis. The first two could be used to test for correctness (module drops, of course), for example, as well as to measure connection establishment performance, etcetera. All are important in general. Nico -- From Alan.Maguire@sun.com Fri Apr 2 13:24:15 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32KOFGK012896 for ; Fri, 2 Apr 2010 13:24:15 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32KOEZE017903 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 2 Apr 2010 13:24:15 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900D07NCFEQ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 13:24:15 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L09003ZJNCDD360@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 13:24:14 -0700 (PDT) Received: from fe-emea-13.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o32KODNd014849 for ; Fri, 02 Apr 2010 20:24:13 +0000 (GMT) Received: from conversion-daemon.fe-emea-13.sun.com by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0900M00N2E0C00@fe-emea-13.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 21:23:49 +0100 (BST) Received: from [192.168.1.8] ([unknown] [86.44.111.72]) by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L0900LGRNBPFY90@fe-emea-13.sun.com>; Fri, 02 Apr 2010 21:23:49 +0100 (BST) Date: Fri, 02 Apr 2010 21:23:50 +0100 From: Alan Maguire Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <20100402164809.GW4225@Sun.COM> Sender: Alan.Maguire@sun.com To: Nicolas Williams Cc: Darren Reed , Adam Leventhal , PSARC-ext@sun.com Message-id: <4BB65256.2080505@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> <20100402164809.GW4225@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 4229 hi Nico On 02/04/2010 17:48, Nicolas Williams wrote: > On Fri, Apr 02, 2010 at 11:53:33AM +0100, Alan Maguire wrote: > >> [...]. I think adding the following >> set of probes to the original proposal should hopefully >> address shortcomings in this area in a manner >> consistent with the connect-* and accept-* probes: >> >> tcp:::close-request >> >> Fires when a FIN segment is sent to a peer, either >> [...] >> >> tcp:::close-established >> >> Fires when an ACK is received to a previously-sent >> FIN segment. This probe firing may simply indicate >> [...] >> >> tcp:::close-accepted >> >> Fires when a FIN segment is received from a peer. >> [...] >> >> tcp:::reset-request >> >> Fires when a RST segment is sent to a peer connection >> [...] >> >> tcp:::reset-accepted >> >> [...] >> > I'd rather see probes named and specified unambiguously after: > > - TCP state transitions (where the name of the probe that fires > represents the new state of a connection and the old state is in some > argument). > > State names should be taken from the standard: CLOSED, LISTEN, > SYN_RCVD, SYN_SENT, EXTABLISHED, CLOSE_WAIT, LAST_ACK, FIN_WAIT_1, > CLOSING, FIN_WAIT_2, TIME_WAIT. > > You might have just one probe, with the old and new states being > arguments. > > Yep - see the tcp:::state-change probe in the original proposal. The above were additional probes, to cover the points Darren raised. > - TCP "signals": locally-initiated close/shutdown, active open > (connect), passive open (listen), receipt of RST, SYN, SYN-ACK, FIN, > FIN-ACK, and ACK of a SYN-ACK, FIN, or FIN-ACK. These should also be > named after the terminology used in the standard. > > Whether a signal is "accepted" or "rejected" (either dropped silently > or responded to via a TCP or ICMP message) should be an argument of > the TCP signals probes. > > You might have just one probe, with the signal being an argument. > (Or two probes, one for locally generated signals and one for remote > signals.) > > These are the tcp:::connect-request, tcp:::connect-established, tcp:::connect-refused (sending a SYN, receiving the final ACK in the three-way handshake, receiving a RST|ACK); tcp:::accept-established and tcp:::accept-refused (received a SYN and responding with SYN|ACK, received a SYN and responding with a RST|ACK); tcp:::close-request, tcp:::close-established and tcp:::close-accepted (sending a FIN, receiving an ACK to a FIN, receiving a FIN). > - TCP options? A probe per-option? Or let the caller parse options > data passed as arguments to the relevant probes? TCP options can > have a huge impact on perfomance (think of SACK, window scaling, > PAWS, ...). Arguably the options that apply to a given connection > should always be available for any probes firing for any connection > in at least ESTABLISHED state. > I've tried to make a certain amount of this state available via the tcpsinfo_t translation of the tcp_t. > - data send/receive, ACKs of data > Yep, see the tcp:::send and tcp:::receive probes. > - window size changes > These are available at the various probe points, either via the tcpsino_t structure (which is a translated version of the tcp_t), or via the translated tcp headers which are arguments to the tcp:::send/receive probes. > - congestion state transitions > These are available at the tcp:::send/tcp:::receive probe points by examining the tcp flags and headers, but it might be useful to add tcp_cwnd to the tcpsinfo_t translation of the tcp_t. > - PMTUD events > > There's already a SDT probe (tcp_update_pmtu) for PATH MTU update events, but again it might be useful to make the MSS is available via the tcpsinfo_t translator of the tcp_t. > The last five are probably crucial for performance analysis. The first > two could be used to test for correctness (module drops, of course), for > example, as well as to measure connection establishment performance, > etcetera. All are important in general. > > Understood. I think it would be a good idea to augment the fields in the tcpsinfo_t with the additional information. Thanks! Alan From darren.reed@oracle.com Mon Apr 5 23:27:36 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o366RaR2005073 for ; Mon, 5 Apr 2010 23:27:36 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o366RVWA062816; Tue, 6 Apr 2010 00:27:34 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0F00M0HZ9XAO00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Apr 2010 23:27:33 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0F00HDGZ9W7OE0@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Apr 2010 23:27:32 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o366RVbF014574; Tue, 06 Apr 2010 06:27:31 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o365EmjN002923; Tue, 06 Apr 2010 06:27:29 +0000 (GMT) Received: from abhmt004.oracle.com by acsmt354.oracle.com with ESMTP id 149188791270535196; Mon, 05 Apr 2010 23:26:36 -0700 Received: from [129.145.154.61] (/129.145.154.61) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Apr 2010 23:26:36 -0700 Date: Mon, 05 Apr 2010 23:26:35 -0700 From: Darren Reed Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB5CCAD.9090503@sun.com> To: Alan Maguire Cc: Adam Leventhal , PSARC-ext@sun.com Message-id: <4BBAD41B.2020907@oracle.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_sxdExUwYjAzxLOsK6N0BCg)" X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BBAD452.0030:SCFMA4539814,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 19768 This is a multi-part message in MIME format. --Boundary_(ID_sxdExUwYjAzxLOsK6N0BCg) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT Hi Alan, You seem to have picked up the ball and run with it quite well, so I think what PSARC needs now is to see a new proposal with all of the additions and other changes that have been discussed. On structuring the document, my thoughts on this are as follows: 1) you're doing design and architecture work that should be able to be picked up by others and used to structure further transport layer probes, so fleshing out the proposal some more could be beneficial in the future to those that follow and want to implement dtrace probes for future networking protocols; 2) now that you've expanded on each of the two protocols, you might like to think about whether it should be one PSARC case or two (I don't think anyone would mind if a newer one had the same timeout as the older one). Feel free to ignore this if you like :) 3) as an exercise for yourself, consider how a new protocol such as RDP (Reliable Datagram Protocl) or ICMP would be mapped into dtrace using what you're introducing as the basis of a design for them. 4) you might like to think about whether or not the probes being introduced fit into different categories. Two such categories might be socket and protocol. For example, the TCP state changes are effectively invisible to the socket and represent internal operations of the protocol. Probes such as socket-open and socket-closed are at a completely different level of abstraction to the TCP state change one yet they're part of the same dtrace provider. I suppose (4) calls into question whether it is the correct architecture to have socket probes per protocol or whether there should be a socket dtrace provider. What are the trade- offs involved in pushing the socket probes into a socket provider vs a protocol provider? I don't want to ask to redesign what's been proposed but I do think there needs to be more thought given to how dtrace interacts with sockets and network protocols than just putting some probes in where we think it will be interesting and help us achieve specific goals. For example, should we be reviewing socket:tcp::open and socket:udp::closed rather than tcp:::socket-open and udp:::socket-closed? Does every protocol need to introduce its own ":::socket-open" and ":::socket-closed" or should it be enough to have just "socket:::open" and "socket:::closed"? Are there elements of the design and how you're using that implementation that would suffer from a different approach to dtrace for TCP and UDP sockets? These are my questions to you and the folks you're working with on this... When I look at the original proposal, I see some interesting work but I don't really see a lot of architecture (just some named interfaces) in a space where there is room for good architecture to be designed and introduced (given time.) I think if the documentation is appropriately structured that we can get some real architecture out of it ;) That's my fast track opinion fodder for this case :) Cheers, Darren On 04/02/10 03:53, Alan Maguire wrote: > Sorry to be following up again, but I think it would > be helpful if I make a specific proposal wrt the > additional probes required, especially the tcp > close-related probes. I think adding the following > set of probes to the original proposal should hopefully > address shortcomings in this area in a manner > consistent with the connect-* and accept-* probes: > > tcp:::close-request > > Fires when a FIN segment is sent to a peer, either > as initiation of a graceful shutdown or in response > to a shutdown request from a peer. > > tcp:::close-established > > Fires when an ACK is received to a previously-sent > FIN segment. This probe firing may simply indicate > a half-close - i.e. the peer connection may not close > from it's side. > > tcp:::close-accepted > > Fires when a FIN segment is received from a peer. > If tcp:::close-accepted and tcp:::close-established > fire, the connection has shut down both sides but > may linger of course. > > tcp:::reset-request > > Fires when a RST segment is sent to a peer connection > to forcibly close (e.g. when we close and SO_LINGER is > set to 0). This is distinguished from a RST|ACK segment, > the sending of which is covered by the tcp:::accept-refused > probe. > > tcp:::reset-accepted > > Fires when a RST segment is received from a peer > connection. > > With these probes in hand, we can ask questions like: > > - how long do connections take to close down? > - which side initiated the close? > - how many half-closed connections are there? > - how many connections close gracefully? how many > close forcefully? > > etc. > > The naming of the probes is consistent with the > connect-* and accept-* probes, and like those > probes they supply the IP/TCP headers associated > with the control segment they represent. The > addition of these probes also means we have > coverage for all the connection control segments > (SYN, FIN, RST) and their related acknowledgments. > > We should also add: > > tcp:::socket-opened > > Fires when a TCP socket is opened. > > tcp:::socket-closed > > Fires when a TCP socket is closed. > > In addition to these, we also have the > tcp:::state-change probe which covers the > "closed" pseudo-state. > > For UDP, as Darren suggests, it would be helpful > to have equivalents for the > accept-refused/connect-refused probes so that > users can detect cases where they attempt > to connect to nonexistent services. I think it > may make sense to stay away from > connect-refused/accept-refused for names as > these terms imply a connection, so I suggest > (the rather unwieldy): > > udp:::port-unreachable-send > > Fires when an ICMP port unreachable message > is sent in response to a UDP datagram heading > for a port with no UDP server. It is true that > an ICMP DTrace provider should cover this event, > but it might be helpful to have a probe point > within the UDP provider for such an event also. > > udp:::port-unreachable-receive > > Fires when an ICMP port unreachable message > is received in response to a UDP datagram. > > Thanks! > > Alan > > On 01/04/2010 20:53, Alan Maguire wrote: >> Apologies, I misstated something below.... >> >> On 01/04/2010 18:30, Alan Maguire wrote: >>> On 01/04/2010 08:33, Darren Reed wrote: >>>> On 03/31/10 22:49, Adam Leventhal wrote: >>>>> Darren, >>>>> >>>>> >>>>>> I'm not sure that enough detail is provided for >>>>>> this to be "automatic" and there is also the problem >>>>>> of the missing UDP probes... see below. >>>>>> >>>>> >>>>> If you scroll down, you'll find the udp provider probes listed. >>>>> >>>>> >>>>>> If we do simple trackng of accept-established and >>>>>> connect-established, we're able to count how many >>>>>> active connections there are but to properly adjust >>>>>> the counter when things close requires tracking >>>>>> state-change and understanding enough of TCP >>>>>> to know how to adjust your stats correctly. >>>>>> >>> True. With respect to tcp close issues, I guess there's >>> a few ways to handle close events: >>> >>> 1. a socket-oriented close probe "tcp:::socket-close" (and >>> for consistency tcp:::socket-open). Adding these seems >>> reasonable to me, especially given the fact that it would >>> be consistent with the udp provider, as you note below. >>> >>> 2. a close probe in the style of the connect-* and >>> accept-* probes - "tcp:::connection-closed" perhaps? >>> This could fire when tcp reaches the fictional >>> closed state for one of the following reasons: >>> >>> - when the TIME_WAIT timeout on a closing connection >>> has expired >>> - when we receive the LAST_ACK for a passive close >>> - when we time out in SYN_SENT state >>> - when we send or receive a RST segment aborting >>> the connection, or we receive a RST|ACK. >>> >>> 3. a state-change probe for the closed state >>> (tcp:::state-change where the new state argument is >>> "state-closed"). This was discussed a while back, and the >>> consensus was not to use the closed state since it is described >>> as fictional in RFC793. Jim describes this here: >>> >>> http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html >>> >>> So my proposal would be to add 1 and 2 above to the >>> provider specification. Does this seem reasonable? >>> >> Actually, we already have 3 - tcp:::state-change where >> the new state is "state-closed". Apologies for the confusion - >> I was writing from memory. So the question is if we should >> add 1 and 2 I guess. >> >> Alan > --Boundary_(ID_sxdExUwYjAzxLOsK6N0BCg) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Hi Alan,

You seem to have picked up the ball and run with it
quite well, so I think what PSARC needs now is to see
a new proposal with all of the additions and other
changes that have been discussed.

On structuring the document, my thoughts on this are
as follows:

1) you're doing design and architecture work that
   should be able to be picked up by others and used
   to structure further transport layer probes, so
   fleshing out the proposal some more could be
   beneficial in the future to those that follow and want
   to implement dtrace probes for future networking
   protocols;

2) now that you've expanded on each of the two protocols,
   you might like to think about whether it should be one
   PSARC case or two (I don't think anyone would mind
   if a newer one had the same timeout as the older one).
   Feel free to ignore this if you like :)

3) as an exercise for yourself, consider how a new protocol
   such as RDP (Reliable Datagram Protocl) or ICMP would
   be mapped into dtrace using what you're introducing as
   the basis of a design for them.

4) you might like to think about whether or not the
   probes being introduced fit into different categories.
   Two such categories might be socket and protocol. For
   example, the TCP state changes are effectively invisible
   to the socket and represent internal operations of the
   protocol. Probes such as socket-open and socket-closed
   are at a completely different level of abstraction to
   the TCP state change one yet they're part of the same
   dtrace provider.

I suppose (4) calls into question whether it is the correct
architecture to have socket probes per protocol or whether
there should be a socket dtrace provider. What are the trade-
offs involved in pushing the socket probes into a socket
provider vs a protocol provider? I don't want to ask to
redesign what's been proposed but I do think there needs to
be more thought given to how dtrace interacts with sockets
and network protocols than just putting some probes in where
we think it will be interesting and help us achieve specific
goals. For example, should we be reviewing socket:tcp::open
and socket:udp::closed rather than tcp:::socket-open and
udp:::socket-closed? Does every protocol need to introduce
its own ":::socket-open" and ":::socket-closed" or should
it be enough to have just "socket:::open" and "socket:::closed"?
Are there elements of the design and how you're using that
implementation that would suffer from a different approach
to dtrace for TCP and UDP sockets? These are my questions
to you and the folks you're working with on this...

When I look at the original proposal, I see some interesting
work but I don't really see a lot of architecture (just some
named interfaces) in a space where there is room for good
architecture to be designed and introduced (given time.)
I think if the documentation is appropriately structured
that we can get some real architecture out of it ;)


That's my fast track opinion fodder for this case :)

Cheers,
Darren


On 04/02/10 03:53, Alan Maguire wrote:
Sorry to be following up again, but I think it would
be helpful if I make a specific proposal wrt the
additional probes required, especially the tcp
close-related probes. I think adding the following
set of probes to the original proposal should hopefully
address shortcomings in this area in a manner
consistent with the connect-* and accept-* probes:

tcp:::close-request

Fires when a FIN segment is sent to a peer, either
as initiation of a graceful shutdown or in response
to a shutdown request from a peer.

tcp:::close-established

Fires when an ACK is received to a previously-sent
FIN segment. This probe firing may simply indicate
a half-close - i.e. the peer connection may not close
from it's side.

tcp:::close-accepted

Fires when a FIN segment is received from a peer.
If tcp:::close-accepted and tcp:::close-established
fire, the connection has shut down both sides but
may linger of course.

tcp:::reset-request

Fires when a RST segment is sent to a peer connection
to forcibly close (e.g. when we close and SO_LINGER is
set to 0). This is distinguished from a RST|ACK segment,
the sending of which is covered by the tcp:::accept-refused
probe.

tcp:::reset-accepted

Fires when a RST segment is received from a peer
connection.

With these probes in hand, we can ask questions like:

- how long do connections take to close down?
- which side initiated the close?
- how many half-closed connections are there?
- how many connections close gracefully? how many
close forcefully?

etc.

The naming of the probes is consistent with the
connect-* and accept-* probes, and like those
probes they supply the IP/TCP headers associated
with the control segment they represent. The
addition of these probes also means we have
coverage for all the connection control segments
(SYN, FIN, RST) and their related acknowledgments.

We should also add:

tcp:::socket-opened

Fires when a TCP socket is opened.

tcp:::socket-closed

Fires when a TCP socket is closed.

In addition to these, we also have the
tcp:::state-change probe which covers the
"closed" pseudo-state.

For UDP, as Darren suggests, it would be helpful
to have equivalents for the
accept-refused/connect-refused probes so that
users can detect cases where they attempt
to connect to nonexistent services. I think it
may make sense to stay away from
connect-refused/accept-refused for names as
these terms imply a connection, so I suggest
(the rather unwieldy):

udp:::port-unreachable-send

Fires when an ICMP port unreachable message
is sent in response to a UDP datagram heading
for a port with no UDP server. It is true that
an ICMP DTrace provider should cover this event,
but it might be helpful to have a probe point
within the UDP provider for such an event also.

udp:::port-unreachable-receive

Fires when an ICMP port unreachable message
is received in response to a UDP datagram.

Thanks!

Alan

On 01/04/2010 20:53, Alan Maguire wrote:
Apologies, I misstated something below....

On 01/04/2010 18:30, Alan Maguire wrote:
On 01/04/2010 08:33, Darren Reed wrote:
On 03/31/10 22:49, Adam Leventhal wrote: 
Darren,

  
I'm not sure that enough detail is provided for
this to be "automatic" and there is also the problem
of the missing UDP probes... see below.
    

If you scroll down, you'll find the udp provider probes listed.

  
If we do simple trackng of accept-established and
connect-established, we're able to count how many
active connections there are but to properly adjust
the counter when things close requires tracking
state-change and understanding enough of TCP
to know how to adjust your stats correctly.
True. With respect to tcp close issues, I guess there's
a few ways to handle close events:

1. a socket-oriented close probe "tcp:::socket-close" (and
for consistency tcp:::socket-open). Adding these seems
reasonable to me, especially given the fact that it would
be consistent with the udp provider, as you note below.

2. a close probe in the style of the connect-* and
accept-* probes - "tcp:::connection-closed" perhaps?
This could fire when tcp reaches the fictional
closed state for one of the following reasons:

- when the TIME_WAIT timeout on a closing connection
has expired
- when we receive the LAST_ACK for a passive close
- when we time out in SYN_SENT state
- when we send or receive a RST segment aborting
the connection, or we receive a RST|ACK.

3. a state-change probe for the closed state
(tcp:::state-change where the new state argument is
"state-closed"). This  was discussed a while back, and the
consensus was not to use the closed state since it is described
as fictional in RFC793. Jim describes this here:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-September/002456.html

So my proposal would be to add 1 and 2 above to the
provider specification. Does this seem reasonable?
Actually, we already have 3 - tcp:::state-change where
the new state is "state-closed". Apologies for the confusion -
I was writing from memory. So the question is if we should
add 1 and 2 I guess.

Alan


--Boundary_(ID_sxdExUwYjAzxLOsK6N0BCg)-- From kacheong.poon@sun.com Wed Apr 7 02:15:50 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o379FoXm005536 for ; Wed, 7 Apr 2010 02:15:50 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o379FlLs003578; Wed, 7 Apr 2010 02:15:48 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0I00L0J1QC4H00@brm-avmta-1.central.sun.com>; Wed, 07 Apr 2010 03:15:48 -0600 (MDT) Received: from jurassic.Eng.Sun.COM ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0I007SK1QBWO90@brm-avmta-1.central.sun.com>; Wed, 07 Apr 2010 03:15:47 -0600 (MDT) Received: from [10.7.251.223] (punchin-kcpoon.SFBay.Sun.COM [10.7.251.223]) by jurassic.Eng.Sun.COM (8.14.4+Sun/8.14.4) with ESMTP id o379Fjp3551947 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 07 Apr 2010 02:15:47 -0700 (PDT) Date: Wed, 07 Apr 2010 17:15:45 +0800 From: Kacheong Poon Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BB5CCAD.9090503@sun.com> To: Alan Maguire Cc: PSARC-ext@sun.com Message-id: <4BBC4D41.9050207@sun.com> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.9) Gecko/20100318 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 5816 On 04/ 2/10 06:53 PM, Alan Maguire wrote: > Sorry to be following up again, but I think it would > be helpful if I make a specific proposal wrt the > additional probes required, especially the tcp > close-related probes. I think adding the following > set of probes to the original proposal should hopefully > address shortcomings in this area in a manner > consistent with the connect-* and accept-* probes: I'd recommend the project team not to add the following probes. Firstly, they are quite confusing. For example, I assume the intention of close-request is that an app using socket does a close(). But note that TCP can send a FIN when an app calls shutdown(SHUT_WR), which has nothing to do with close(). And TCP does not have a "shutdown request from a peer" mechanism. Secondly, the probes seem to be tied to socket calls. But Solaris TCP also supports TLI. It does not seem right to have API specific probes in the TCP level. If the project team wants to track API specific calls, I'd suggest that the team adds sockfs or TLI level probes instead of having them in TCP. And what are their intended usage? I don't understand why they are needed to check for those conditions in their descriptions. The send/receive/state-change probes should be enough. I think one reason for having the original connect-*/accept-* calls is that the project wants to have the packet info which triggers the state change to be passed to the probe. Checking the examples in http://wikis.sun.com/display/DTrace/tcp+Provider I don't think the packet info is necessary. The scripts can be written using the tcpsinfo_t. The missing piece is the IP addresses, which can be added to tcpsinfo_t or can be found in the tcp_t (returned in tcps_addr). For those folks who must check the packet which triggers a state change, I think those folks should also understand TCP (the protocol, not our implementation) enough that they can use both the receive probe and the state-change probes to capture the packet which triggers a state change. I don't think the packet info is needed in general to require adding special probes. What we need to have is good examples showing what can be done. For example, we can have examples showing how connect-*/accept-* probes can be written using send/receive/state-change probes. > tcp:::close-request > > Fires when a FIN segment is sent to a peer, either > as initiation of a graceful shutdown or in response > to a shutdown request from a peer. > > tcp:::close-established > > Fires when an ACK is received to a previously-sent > FIN segment. This probe firing may simply indicate > a half-close - i.e. the peer connection may not close > from it's side. > > tcp:::close-accepted > > Fires when a FIN segment is received from a peer. > If tcp:::close-accepted and tcp:::close-established > fire, the connection has shut down both sides but > may linger of course. > > tcp:::reset-request > > Fires when a RST segment is sent to a peer connection > to forcibly close (e.g. when we close and SO_LINGER is > set to 0). This is distinguished from a RST|ACK segment, > the sending of which is covered by the tcp:::accept-refused > probe. > > tcp:::reset-accepted > > Fires when a RST segment is received from a peer > connection. > > With these probes in hand, we can ask questions like: > > - how long do connections take to close down? > - which side initiated the close? > - how many half-closed connections are there? > - how many connections close gracefully? how many > close forcefully? > > etc. > > The naming of the probes is consistent with the > connect-* and accept-* probes, and like those > probes they supply the IP/TCP headers associated > with the control segment they represent. The > addition of these probes also means we have > coverage for all the connection control segments > (SYN, FIN, RST) and their related acknowledgments. > > We should also add: > > tcp:::socket-opened > > Fires when a TCP socket is opened. > > tcp:::socket-closed > > Fires when a TCP socket is closed. Besides the fact that their names are API specific, what are their intended usage? What is the interesting fact that we need a probe to fire when a TCP end point (socket or TLI) is created? If that end point does not send/receive anything, I guess from TCP's point of view, it is not interesting. > In addition to these, we also have the > tcp:::state-change probe which covers the > "closed" pseudo-state. I'd suggest that that project team creates only the send/receive/state-change probes in the TCP provider. > For UDP, as Darren suggests, it would be helpful > to have equivalents for the > accept-refused/connect-refused probes so that > users can detect cases where they attempt > to connect to nonexistent services. I think it > may make sense to stay away from > connect-refused/accept-refused for names as > these terms imply a connection, so I suggest > (the rather unwieldy): > > udp:::port-unreachable-send > > Fires when an ICMP port unreachable message > is sent in response to a UDP datagram heading > for a port with no UDP server. It is true that > an ICMP DTrace provider should cover this event, > but it might be helpful to have a probe point > within the UDP provider for such an event also. > > udp:::port-unreachable-receive > > Fires when an ICMP port unreachable message > is received in response to a UDP datagram. I'd recommend the project team not to add the above UDP probes. They are really "after the fact." For example, the UDP end point can be already gone when the ICMP message is received. As suggested by the team, it is better to have an ICMP provider to catch this kind of events. So I'd suggest that the team only creates the send/receive probes in the UDP provider. -- K. Poon. kacheong.poon@sun.com From darren.reed@oracle.com Wed Apr 7 02:40:54 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o379esFQ005593 for ; Wed, 7 Apr 2010 02:40:54 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o379eoUI026121; Wed, 7 Apr 2010 04:40:52 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0I0040L2W30800@nwk-avmta-1.sfbay.Sun.COM>; Wed, 07 Apr 2010 02:40:51 -0700 (PDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0I00DR72W38110@nwk-avmta-1.sfbay.Sun.COM>; Wed, 07 Apr 2010 02:40:51 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o379eof5026682; Wed, 07 Apr 2010 09:40:50 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o36LwBaA022128; Wed, 07 Apr 2010 09:40:49 +0000 (GMT) Received: from abhmt001.oracle.com by acsmt355.oracle.com with ESMTP id 142355121270633141; Wed, 07 Apr 2010 02:39:01 -0700 Received: from [129.145.154.61] (/129.145.154.61) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 07 Apr 2010 02:39:00 -0700 Date: Wed, 07 Apr 2010 02:38:59 -0700 From: Darren Reed Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BBC4D41.9050207@sun.com> To: Kacheong Poon Cc: Alan Maguire , PSARC-ext@sun.com Message-id: <4BBC52B3.6040003@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4BBC5321.0069:SCFMA4539814,ss=1,fgs=0 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> <4BBC4D41.9050207@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 2694 On 04/07/10 02:15, Kacheong Poon wrote: > On 04/ 2/10 06:53 PM, Alan Maguire wrote: >> Sorry to be following up again, but I think it would >> be helpful if I make a specific proposal wrt the >> additional probes required, especially the tcp >> close-related probes. I think adding the following >> set of probes to the original proposal should hopefully >> address shortcomings in this area in a manner >> consistent with the connect-* and accept-* probes: > > > I'd recommend the project team not to add the following > probes. Firstly, they are quite confusing. For example, > I assume the intention of close-request is that an app > using socket does a close(). But note that TCP can send > a FIN when an app calls shutdown(SHUT_WR), which has > nothing to do with close(). And TCP does not have a > "shutdown request from a peer" mechanism. Secondly, the > probes seem to be tied to socket calls. But Solaris TCP > also supports TLI. It does not seem right to have API > specific probes in the TCP level. If the project team > wants to track API specific calls, I'd suggest that the > team adds sockfs or TLI level probes instead of having > them in TCP. Yup, that's along the same lines as me asking whether they should be introducing socket probes instead... ... >> For UDP, as Darren suggests, it would be helpful >> to have equivalents for the >> accept-refused/connect-refused probes so that >> users can detect cases where they attempt >> to connect to nonexistent services. I think it >> may make sense to stay away from >> connect-refused/accept-refused for names as >> these terms imply a connection, so I suggest >> (the rather unwieldy): >> >> udp:::port-unreachable-send >> >> Fires when an ICMP port unreachable message >> is sent in response to a UDP datagram heading >> for a port with no UDP server. It is true that >> an ICMP DTrace provider should cover this event, >> but it might be helpful to have a probe point >> within the UDP provider for such an event also. >> >> udp:::port-unreachable-receive >> >> Fires when an ICMP port unreachable message >> is received in response to a UDP datagram. > > > I'd recommend the project team not to add the above > UDP probes. They are really "after the fact." For > example, the UDP end point can be already gone when > the ICMP message is received. As suggested by the > team, it is better to have an ICMP provider to catch > this kind of events. So I'd suggest that the team > only creates the send/receive probes in the UDP > provider. Do you think that it might be useful to have a probe that is triggered when an ICMP error packet arrives for an open socket? (But not as an ICMP protocol probe.) Darren From kacheong.poon@sun.com Wed Apr 7 02:57:00 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o379v0KQ005958 for ; Wed, 7 Apr 2010 02:57:00 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o379uwmN029913; Wed, 7 Apr 2010 02:56:58 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0I008113MYGT00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 07 Apr 2010 02:56:58 -0700 (PDT) Received: from jurassic.Eng.Sun.COM ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0I00DRG3MX7K40@nwk-avmta-1.sfbay.Sun.COM>; Wed, 07 Apr 2010 02:56:57 -0700 (PDT) Received: from [10.7.251.223] (punchin-kcpoon.SFBay.Sun.COM [10.7.251.223]) by jurassic.Eng.Sun.COM (8.14.4+Sun/8.14.4) with ESMTP id o379ut5C554904 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 07 Apr 2010 02:56:57 -0700 (PDT) Date: Wed, 07 Apr 2010 17:56:53 +0800 From: Kacheong Poon Subject: Re: DTrace TCP and UDP providers [PSARC/2010/106 Self Review] In-reply-to: <4BBC52B3.6040003@oracle.com> To: Darren Reed Cc: Alan Maguire , PSARC-ext@sun.com Message-id: <4BBC56E5.3030306@sun.com> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003311736.o2VHanuW156258@zday.sfbay.sun.com> <4BB42F8A.2030901@oracle.com> <2E77E4A0-2671-4378-B81C-BC9015F6E3A2@eng.sun.com> <4BB44C30.9040105@oracle.com> <4BB4D83F.201@sun.com> <4BB4F9D4.4010207@sun.com> <4BB5CCAD.9090503@sun.com> <4BBC4D41.9050207@sun.com> <4BBC52B3.6040003@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.9) Gecko/20100318 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 600 On 04/ 7/10 05:38 PM, Darren Reed wrote: > Do you think that it might be useful to have a probe that > is triggered when an ICMP error packet arrives for an open > socket? (But not as an ICMP protocol probe.) In TCP/UDP provider? Then we need to have such probes in each provider to be consistent. I guess the ICMP provider can return the conn_t (or using cs_cid) in this case when the upper layer protocol end point is still around. And this works for all transport providers. The conn_t (or cs_cid) can be used to tie those probes together. -- K. Poon. kacheong.poon@sun.com From alan.maguire@oracle.com Sun May 2 02:34:06 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o429Y6cY024684 for ; Sun, 2 May 2010 02:34:06 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o429Y65V000087 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sun, 2 May 2010 02:34:06 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1S00001D8UWA00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 02 May 2010 03:34:06 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1S006ISD8T8V40@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 02 May 2010 03:34:05 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o429Y5Gw020029 for ; Sun, 02 May 2010 09:34:05 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o429Y473006070 for ; Sun, 02 May 2010 09:34:04 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt354.oracle.com with ESMTP id 207601961272792747; Sun, 02 May 2010 02:32:27 -0700 Received: from [192.168.1.1] (/86.44.118.66) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 02 May 2010 02:32:27 -0700 Date: Sun, 02 May 2010 10:32:27 +0100 From: Alan Maguire Subject: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] To: PSARC-ext Message-id: <4BDD46AB.8090107@oracle.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_5nVhHFnk4O87VURel/048Q)" X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4BDD470C.016E:SCFMA4539814,ss=1,fgs=0 User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 13926 This is a multi-part message in MIME format. --Boundary_(ID_5nVhHFnk4O87VURel/048Q) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT As per the subject line, the updated spec is attached for the above fasttrack. In the interim, the project team have worked offline with Kacheong and Darren to address their concerns. We have also incorporated SACK, PMTU and congestion window info into the tcpsinfo_t at Nico's suggestion. Thanks! Alan --Boundary_(ID_5nVhHFnk4O87VURel/048Q) Content-type: text/plain; name=PSARC_2010_106_tcp_udp_provider_spec.txt Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=PSARC_2010_106_tcp_udp_provider_spec.txt Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2010 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: DTrace TCP/UDP Providers 1.2. Name of Document Author/Supplier: Author: Brendan Gregg/Alan Maguire 1.3 Date of This Document: 1 May, 2010 4. Technical Description A. INTRODUCTION This case adds DTrace 'tcp' and 'udp' providers with probes for send and receive events. These providers cover the TCP and UDP protocol implementations in OpenSolaris respectively. In addition the tcp provider contains probes for TCP state machine transitions and events relating to TCP connection establishment. We initially focus on connection establishment events in TCP since there are solid use cases for those probes (connection latency, first-byte latency, port scan detection etc). In the future, if there is similar demand for connection termination probes we will provide these also. These providers are intended for use by customers for network observability and troubleshooting, and this work represents the second and third components of a suite of planned providers for the network stack. The first was described in PSARC/2008/302 DTrace IP Provider. The provider design described here is intended to set an architectural precendent for similar future providers (e.g. SCTP, ICMP, RDP) in terms of probe names: - :::send, :::receive, and :::state-change (the latter for stateful protocols) should be used where possible for transmit, receive and state transition events ...and probe arguments: - For consistency, future networking providers should attempt to adhere to a similar argument order: arg0: pktinfo_t * arg1: csinfo_t * arg2: ipinfo_t * arg3: protocol state-specific sinfo_t (e.g. tcpsinfo_t *, sctpsinfo_t *) arg4: protocol-specific header info_t (e.g. tcpinfo_t *, sctpinfo_t *) B. TCP PROBE DESCRIPTION This work will introduce the following probes for the 'tcp' provider: tcp:::send TCP transmits a segment. tcp:::receive TCP receives a segment. tcp:::state-change TCP state machine transition. Previous state is noted in the tcplsinfo_t * probe argument. The tcpinfo_t * and ipinfo_t * arguments are NULL. tcp:::connect-request A TCP active open is initiated by sending an initial SYN segment. The tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers associated with the initial SYN segment sent. tcp:::connect-established This probe fires when either of the following occurs: - A TCP active OPEN succeeds - the initial SYN has been sent and a valid SYN|ACK segment has been received in response. TCP enters the ESTABLISHED state. The tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers associated with the SYN|ACK segment received; or - A simultaneous active OPEN succeeds and a final ACK is received from the peer TCP. TCP has entered the ESTABLISHED state and the tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers of the final ACK received. The common thread in these cases is that an active-OPEN connection is established at this point, in contrast with tcp:::accept-established which fires on passive connection establishment. In both cases above, the TCP segment that is presented via the tcpinfo_t * is the segment that triggers the transition to ESTABLISHED - the received SYN|ACK in the first case and the final ACK segment in the second. tcp:::connect-refused A TCP active OPEN connection attempt was refused by the peer - a RST segment was received in acknowledgment of the initial SYN. The tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers associated with the RST|ACK segment received. tcp:::accept-established A passive open has succeeded - an initial active OPEN initiation SYN has been received, TCP responded with a SYN|ACK and a final ACK has been received. TCP has entered the ESTABLISHED state. The tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers associated with the final ACK segment received. tcp:::accept-refused An incoming SYN has arrived for a destination port with no listening connection, so the connection initiation request is rejected by sending a RST segment ACKing the SYN. The tcpinfo_t * and ipinfo_t * probe arguments represent the TCP and IP headers associated with the RST segment sent. This case postpones addressing connection termination at this point since use cases for such probes have not yet been determined. However, the addition of connection termination probes will be evaluated in the future if use cases demonstrate that such probes would be valuable (the above probes are useful for scenarios such as determining connection latency, first-byte latency, detecting port scans etc). The project team explored using TCP state transitions coupled with predicates instead of introducing the above connection-related probes, however in prototyping out a solution these proved complex and introduced performance problems due to having to probe for a wide set of events and then evaluate a predicate for the narrower set of events of interest - an enabled-probe effect in other words. C. TCP PROBE ARGUMENTS The arguments to these probes are: args[0] pktinfo_t * packet info args[1] csinfo_t * connection state info args[2] ipinfo_t * generic IP info args[3] tcpsinfo_t * TCP state information args[4] tcpinfo_t * TCP header args[5] tcplsinfo_t * Previous TCP state The order and content has been chosen for consistency with other network providers. The ipinfo_t * and tcpinfo_t * will be NULL for tcp:::state-change events, and the tcplsinfo_t * argument is present for the tcp:::state-change probe only. The arguments contain: /* * pktinfo is where packet ID info can be made available for deeper * analysis if packet IDs become supported by the kernel in the future. * The pkt_addr member is currently always NULL. */ typedef struct pktinfo { uintptr_t pkt_addr; } pktinfo_t; /* * csinfo is where connection state info is made available. */ typedef struct csinfo { uintptr_t cs_addr; uint64_t cs_cid; pid_t cs_pid; zoneid_t cs_zoneid; } csinfo_t; Note that we have filled out the csinfo_t with additional information. In PSARC/2008/302 DTrace IP Provider, only the cs_addr was present. Here we take advantage of the fact that the IP datapath refactoring project modified IP such that a transmit attribute structure - ip_xmit_attr_t * - travels with the packet on the outbound path; and on the inbound path, the conn_t contains an ip_xmit_attr_t *, so that when a packet is classified by IP and passed up to UDP, SCTP or TCP, we have access to information about the target process, zone etc. We will also add a new connection id field to the ip_xmit_attr_t. Having such an identifier is extremely useful in DTrace. Future work will ensure this extended csinfo_t is used at the ip:::send probe points also, though significant code refactoring will be required to ensure availability of the ip_xmit_attr_t * at all these points. On the ip:::receive side, it is not possible to make this mapping as the IP packet has not yet been classified (and mapped to a conn_t) when ip:::receive fires. /* * ipinfo contains common IP info for both IPv4 and IPv6. */ typedef struct ipinfo { uint8_t ip_ver; /* IP version (4, 6) */ uint16_t ip_plength; /* payload length */ string ip_saddr; /* source address */ string ip_daddr; /* destination address */ } ipinfo_t; /* * tcpsinfo contains stable TCP details from tcp_t. */ typedef struct tcpsinfo { uintptr tcps_addr; int tcps_local; /* is delivered locally, boolean */ int tcps_active; /* active open (from here), boolean */ uint16_t tcps_lport; /* local port */ uint16_t tcps_rport; /* remote port */ string tcps_laddr; /* local address, as a string */ string tcps_raddr; /* remote address, as a string */ int32_t tcps_state; /* TCP state */ uint32_t tcps_iss; /* initial sequence # */ uint32_t tcps_suna; /* sequence # sent but unacked */ uint32_t tcps_snxt; /* next sequence # to send */ uint32_t tcps_rack; /* sequence # we have acked */ uint32_t tcps_rnxt; /* next sequence # expected */ uint32_t tcps_swnd; /* send window size */ uint32_t tcps_snd_ws; /* send window scaling */ uint32_t tcps_rwnd; /* receive window size */ uint32_t tcps_rcv_ws; /* receive window scaling */ uint32_t tcps_cwnd; /* congestion window */ uint32_t tcps_cwnd_ssthresh; /* threshold for congestion avoidance */ uint32_t tcps_sack_fack; /* SACK sequence # we have acked */ uint32_t tcps_sack_snxt; /* next SACK seq # for retransmission */ uint32_t tcps_rto; /* round-trip timeout, msec */ uint32_t tcps_mss; /* max segment size */ int tcps_retransmit; /* retransmit send event, boolean */ } tcpsinfo_t; The tcpsinfo_t has been expanded to cover SACK-, congestion window- and PMTU-related data, as was suggested. /* * tcpinfo is the TCP header fields. */ typedef struct tcpinfo { uint16_t tcp_sport; /* source port */ uint16_t tcp_dport; /* destination port */ uint32_t tcp_seq; /* sequence number */ uint32_t tcp_ack; /* acknowledgment number */ uint8_t tcp_offset; /* data offset, in bytes */ uint8_t tcp_flags; /* flags */ uint16_t tcp_window; /* window size */ uint16_t tcp_checksum; /* checksum */ uint16_t tcp_urgent; /* urgent data pointer */ tcph_t *tcp_hdr; /* raw TCP header */ } tcpinfo_t; /* * tcpnsinfo provides the new tcp state for state changes. */ typedef struct tcplsinfo { int32_t tcps_state; /* TCP state */ } tcplsinfo_t; D. UDP PROBE DESCRIPTION This work will also introduce the following probes for the 'udp' provider: udp:::send UDP sends a datagram. udp:::receive UDP receives a datagram. E. UDP PROBE ARGUMENTS The arguments to these probes are: args[0] pktinfo_t * packet info args[1] csinfo_t * connection state info args[2] ipinfo_t * generic IP info args[3] udpsinfo_t * UDP state information args[4] udpinfo_t * UDP header /* * udpsinfo contains stable UDP details from udp_t. */ typedef struct udpsinfo { uintptr_t udps_addr; uint16_t upds_lport; /* local port */ uint16_t udps_rport; /* remote port */ string udps_laddr; /* local address, as a string */ string udps_raddr; /* remote address, as a string */ } udpsinfo_t; /* * udpinfo is the UDP header fields. */ typedef struct udpinfo { uint16_t udp_sport; /* source port */ uint16_t udp_dport; /* destination port */ uint16_t udp_length; /* total length */ uint16_t udp_checksum; /* headers + data checksum */ udpha_t *udp_hdr; /* raw UDP header */ } udpinfo_t; F. EXAMPLES # Watch inbound TCP connections by remote address. dtrace -n 'tcp:::accept-established { trace(args[2]->ip_saddr); }' # Inbound TCP connections by destination port summary. dtrace -n 'tcp:::accept-established { @port[args[4]->tcp_dport] = count(); }' # Watch inbound accepted TCP connections by process summary. dtrace -n 'tcp:::accept-established { @cpid[args[1]->cs_pid] = count(); }' # Watch UDP total number of bytes sent/received by process. dtrace -n 'udp:::send,udp:::receive { @bytes[args[1]->cs_pid] = sum(args[4]->udp_length);}' G. REFERENCES The suite of planned providers is described on the following website, which includes demonstrations and source from previous prototypes: http://www.opensolaris.org/os/community/dtrace/NetworkProvider These providers have also been discussed in the past on both dtrace-discuss and networking-discuss: http://www.opensolaris.org/jive/thread.jspa?messageID=57666 http://www.opensolaris.org/jive/thread.jspa?messageID=128518😆 RFC793 http://tools.ietf.org/html/rfc793 H. DOCUMENTATION New chapters will be added to the current Solaris Dynamic Tracing Guide for these proposed providers, and demo scripts will be added to /usr/demo/dtrace. The tcp provider is described here: http://wikis.sun.com/display/DTrace/tcp+Provider ...and the udp provider is described here: http://wikis.sun.com/display/DTrace/udp+Provider I. STABILITY The DTrace internal stability table is described below: Element Name stability Data stability Dependency class Provider Evolving Evolving ISA Module Private Private Unknown Function Private Private Unknown Name Evolving Evolving ISA Arguments Evolving Evolving ISA 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: OS/Net 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open --Boundary_(ID_5nVhHFnk4O87VURel/048Q)-- From ahl@eng.sun.com Mon May 3 09:35:09 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o43GZ963026398 for ; Mon, 3 May 2010 09:35:09 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o43GZ8IP023325 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 3 May 2010 09:35:08 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1U00L27REKTG00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 03 May 2010 09:35:08 -0700 (PDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1U00HPMREINA30@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 03 May 2010 09:35:07 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o43GZ6s4028196 for ; Mon, 03 May 2010 16:35:06 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o43GZ0MM017775 for ; Mon, 03 May 2010 16:35:01 +0000 (GMT) Received: from abhmt013.oracle.com by acsmt355.oracle.com with ESMTP id 231556401272904498; Mon, 03 May 2010 09:34:58 -0700 Received: from qiviut.sf.fishpong.com (/198.144.208.55) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 May 2010 09:34:57 -0700 Date: Mon, 03 May 2010 09:35:02 -0700 From: Adam Leventhal Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BDD46AB.8090107@oracle.com> To: Alan Maguire Cc: PSARC-ext Message-id: <583B8055-62D8-41F3-92F3-5621EC6825C5@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1078) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4BDEFB39.00EE,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> Status: RO Content-Length: 602 Now that the outstanding concerns have been addressed, I've reset the timer for this Wednesday, 5/5. Adam On May 2, 2010, at 2:32 AM, Alan Maguire wrote: > As per the subject line, the updated spec is attached for > the above fasttrack. In the interim, the project team have > worked offline with Kacheong and Darren to address their > concerns. We have also incorporated SACK, PMTU and > congestion window info into the tcpsinfo_t at Nico's > suggestion. Thanks! > > Alan > -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From Nicolas.Williams@oracle.com Tue May 4 10:38:05 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o44Hc5wc013068 for ; Tue, 4 May 2010 10:38:05 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o44Hc4VJ022089 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 10:38:05 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1W00C0LOZHNX00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:38:05 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1W003WWOZGT860@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:38:04 -0600 (MDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o44Hc41j009877 for ; Tue, 04 May 2010 17:38:04 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o44Hc1jo016391 for ; Tue, 04 May 2010 17:38:01 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt355.oracle.com with ESMTP id 213667141272994617; Tue, 04 May 2010 10:36:57 -0700 Received: from oracle.com (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 04 May 2010 10:36:56 -0700 Date: Tue, 04 May 2010 12:36:52 -0500 From: Nicolas Williams Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BDD46AB.8090107@oracle.com> To: Alan Maguire Cc: PSARC-ext Message-id: <20100504173651.GN9429@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4BE05B7B.0004:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 446 On Sun, May 02, 2010 at 10:32:27AM +0100, Alan Maguire wrote: > tcp:::send > > TCP transmits a segment. > > tcp:::receive > > TCP receives a segment. I don't see probes for sending/receiving ACKs; I assume these two fire even for packets that have no data. I also don't see a probe for a SYN being received (well, there's the state change probe, so that's OK). Too bad about the lack of close-related probes, but that's OK too. Nico -- From Nicolas.Williams@oracle.com Tue May 4 10:45:18 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o44HjI7J013166 for ; Tue, 4 May 2010 10:45:18 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o44HjCVr065359 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 11:45:18 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1W0040FPBHNY00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 10:45:17 -0700 (PDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1W00MQ0PBHMP50@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 10:45:17 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o44HjHag000212 for ; Tue, 04 May 2010 17:45:17 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o43MiAse029474 for ; Tue, 04 May 2010 17:45:16 +0000 (GMT) Received: from abhmt009.oracle.com by acsmt354.oracle.com with ESMTP id 213689721272995096; Tue, 04 May 2010 10:44:56 -0700 Received: from oracle.com (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 04 May 2010 10:44:55 -0700 Date: Tue, 04 May 2010 12:44:49 -0500 From: Nicolas Williams Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BDD46AB.8090107@oracle.com> To: Alan Maguire Cc: PSARC-ext Message-id: <20100504174449.GO9429@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090205.4BE05D2C.0116:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 909 It's probably too late for this case, but if so, then something to keep in mind for a future case: IPsec latching (see ipsec_latch_t, in $SRC/uts/common/inet/ip.h). This could either be subsumed into ipinfo_t or, better, added as one more probe argument, both on receive and on send probes. IPsec latching is a mechanism by which Solaris ensures that a connection uses the same IPsec protection parameters throughout its lifetime. IPsec latches are as ancilliary data sent up and down the stack along with the packet. Having IPsec latch info in these probes would allow one to observe the effect of using the IP_SEC_OPT socket option, as well as the effect of changing IPsec configuration. This will be useful mainly for observability. You could argue that IPsec support should come later, and I won't mind :) Sorry I missed this earlier. And thanks for adding the SACK and PMTU information. Nico -- From ahl@eng.sun.com Tue May 4 11:00:02 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o44I02qx013669 for ; Tue, 4 May 2010 11:00:02 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o44I026O003494 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 11:00:02 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1W00E29Q02SZ00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 12:00:02 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1W003KIQ00TD70@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 12:00:01 -0600 (MDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o44I005r019415 for ; Tue, 04 May 2010 18:00:00 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o44FRu7K031196 for ; Tue, 04 May 2010 17:59:59 +0000 (GMT) Received: from abhmt009.oracle.com by acsmt355.oracle.com with ESMTP id 235903641272995891; Tue, 04 May 2010 10:58:11 -0700 Received: from [192.168.3.209] (/198.144.208.55) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 04 May 2010 10:58:11 -0700 Date: Tue, 04 May 2010 10:57:58 -0700 From: Adam Leventhal Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <20100504174449.GO9429@oracle.com> To: Nicolas Williams Cc: Alan Maguire , PSARC-ext Message-id: <48B33981-9B19-4A85-8E7D-B2D15F4EB058@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1078) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4BE0609F.00F0,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> <20100504174449.GO9429@oracle.com> Status: RO Content-Length: 1211 Hey Nico, Those would be solid improvements; we'd like to defer them to a future case enhancing these providers. Adam On May 4, 2010, at 10:44 AM, Nicolas Williams wrote: > It's probably too late for this case, but if so, then something to keep > in mind for a future case: IPsec latching (see ipsec_latch_t, in > $SRC/uts/common/inet/ip.h). This could either be subsumed into ipinfo_t > or, better, added as one more probe argument, both on receive and on > send probes. > > IPsec latching is a mechanism by which Solaris ensures that a connection > uses the same IPsec protection parameters throughout its lifetime. > IPsec latches are as ancilliary data sent up and down the stack along > with the packet. > > Having IPsec latch info in these probes would allow one to observe the > effect of using the IP_SEC_OPT socket option, as well as the effect of > changing IPsec configuration. This will be useful mainly for > observability. You could argue that IPsec support should come later, > and I won't mind :) > > Sorry I missed this earlier. And thanks for adding the SACK and PMTU > information. > > Nico > -- > -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From alan.maguire@oracle.com Tue May 4 11:16:56 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o44IGu96013967 for ; Tue, 4 May 2010 11:16:56 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o44IGsLO006566 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 13:16:56 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1W00617QS7IU00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:16:55 -0700 (PDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1W00M2MQS7MR60@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:16:55 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o44IGsQ4012896 for ; Tue, 04 May 2010 18:16:55 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o44G1rKu022823 for ; Tue, 04 May 2010 18:16:52 +0000 (GMT) Received: from abhmt006.oracle.com by acsmt354.oracle.com with ESMTP id 213801751272996981; Tue, 04 May 2010 11:16:21 -0700 Received: from [192.168.1.1] (/86.44.123.94) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 04 May 2010 11:16:20 -0700 Date: Tue, 04 May 2010 19:16:22 +0100 From: Alan Maguire Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <20100504173651.GN9429@oracle.com> To: Nicolas Williams Cc: PSARC-ext Message-id: <4BE06476.5060804@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BE06496.0096:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> <20100504173651.GN9429@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 665 On 04/05/2010 18:36, Nicolas Williams wrote: > On Sun, May 02, 2010 at 10:32:27AM +0100, Alan Maguire wrote: > >> tcp:::send >> >> TCP transmits a segment. >> >> tcp:::receive >> >> TCP receives a segment. >> > I don't see probes for sending/receiving ACKs; I assume these two fire > even for packets that have no data. > > They do indeed. > I also don't see a probe for a SYN being received (well, there's the > state change probe, so that's OK). tcp:::send,receive fire for both data and control segments (SYN/ACK/FIN). I've stated that explicitly in the DTrace wiki tcp provider doc, but probably should have clarified it here too. Thanks! Alan From Nicolas.Williams@oracle.com Tue May 4 11:20:06 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o44IK5Xx014051 for ; Tue, 4 May 2010 11:20:05 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o44IK3Wp008226 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 13:20:05 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1W0082FQXGDY00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:20:04 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1W002DFQXEUK60@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 11:20:02 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o44IK1cR013601 for ; Tue, 04 May 2010 18:20:02 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o44IJw1P013628 for ; Tue, 04 May 2010 18:19:58 +0000 (GMT) Received: from abhmt018.oracle.com by acsmt353.oracle.com with ESMTP id 235996931272997163; Tue, 04 May 2010 11:19:23 -0700 Received: from oracle.com (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 04 May 2010 11:19:23 -0700 Date: Tue, 04 May 2010 13:19:18 -0500 From: Nicolas Williams Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BE06476.5060804@oracle.com> To: Alan Maguire Cc: PSARC-ext Message-id: <20100504181918.GQ9429@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BE06551.0120:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> <20100504173651.GN9429@oracle.com> <4BE06476.5060804@oracle.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 758 On Tue, May 04, 2010 at 07:16:22PM +0100, Alan Maguire wrote: > On 04/05/2010 18:36, Nicolas Williams wrote: > >On Sun, May 02, 2010 at 10:32:27AM +0100, Alan Maguire wrote: > >>tcp:::send > >> > >> TCP transmits a segment. > >> > >>tcp:::receive > >> > >> TCP receives a segment. > >I don't see probes for sending/receiving ACKs; I assume these two fire > >even for packets that have no data. > > > They do indeed. > >I also don't see a probe for a SYN being received (well, there's the > >state change probe, so that's OK). > tcp:::send,receive fire for both data and control segments > (SYN/ACK/FIN). I've stated that explicitly in the DTrace wiki > tcp provider doc, but probably should have clarified it here too. > Thanks! Perfect. Looks good to me! From Darren.Reed@sun.com Tue May 4 18:59:50 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o451xomM023958 for ; Tue, 4 May 2010 18:59:50 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o451xmvb022093 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 4 May 2010 18:59:48 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1X00D01C7OZY00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 18:59:48 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1X00HTZC7NM0B0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 04 May 2010 18:59:48 -0700 (PDT) Received: from fe-emea-13.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o451xkYA018161 for ; Wed, 05 May 2010 01:59:47 +0000 (GMT) Received: from conversion-daemon.fe-emea-13.sun.com by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L1X00H00C049X00@fe-emea-13.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 05 May 2010 02:59:46 +0100 (BST) Received: from mintslice.sfbay.sun.com ([unknown] [129.146.106.55]) by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L1X00B02C7KJL50@fe-emea-13.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 05 May 2010 02:59:46 +0100 (BST) Date: Tue, 04 May 2010 19:00:00 -0700 From: Darren Reed Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BDD46AB.8090107@oracle.com> Sender: Darren.Reed@sun.com To: Alan Maguire Cc: PSARC-ext Message-id: <4BE0D120.6010707@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BDD46AB.8090107@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-GB; rv:1.9.1.5) Gecko/20091206 Thunderbird/3.0 Status: RO Content-Length: 820 On 2/05/10 02:32 AM, Alan Maguire wrote: > As per the subject line, the updated spec is attached for > the above fasttrack. In the interim, the project team have > worked offline with Kacheong and Darren to address their > concerns. We have also incorporated SACK, PMTU and > congestion window info into the tcpsinfo_t at Nico's > suggestion. Thanks! Thanks, Alan, that looks great. I do have one question though (but only indirectly about this case)... When systems are under high load... - will dtrace buffers be expanded to meet demand? - will performance be slowed to ensure there is no loss with respect to dtrace probes? - is there any scope for being able to communicate that dtrace has missed a packet (or two), should that happen if all of the buffers are full? (This is only important for TCP) Darren From alan.maguire@oracle.com Wed May 5 00:25:36 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o457PaAl018754 for ; Wed, 5 May 2010 00:25:36 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o457PYMG025390; Wed, 5 May 2010 00:25:34 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1X00A0PRAMK500@brm-avmta-1.central.sun.com>; Wed, 05 May 2010 01:25:34 -0600 (MDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1X00JJ7RAL7350@brm-avmta-1.central.sun.com>; Wed, 05 May 2010 01:25:33 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o457PWCJ029525; Wed, 05 May 2010 07:25:32 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o453tinV032647; Wed, 05 May 2010 07:25:32 +0000 (GMT) Received: from abhmt008.oracle.com by acsmt353.oracle.com with ESMTP id 237763241273044295; Wed, 05 May 2010 00:24:55 -0700 Received: from [192.168.1.1] (/86.44.123.94) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 05 May 2010 00:24:55 -0700 Date: Wed, 05 May 2010 08:24:58 +0100 From: Alan Maguire Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BE0D120.6010707@Sun.COM> To: Darren Reed Cc: PSARC-ext Message-id: <4BE11D4A.3030405@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BE11D6C.00A5:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> <4BE0D120.6010707@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 Status: RO Content-Length: 2856 On 05/05/2010 03:00, Darren Reed wrote: > On 2/05/10 02:32 AM, Alan Maguire wrote: >> As per the subject line, the updated spec is attached for >> the above fasttrack. In the interim, the project team have >> worked offline with Kacheong and Darren to address their >> concerns. We have also incorporated SACK, PMTU and >> congestion window info into the tcpsinfo_t at Nico's >> suggestion. Thanks! > > Thanks, Alan, that looks great. > > I do have one question though (but only indirectly about this case)... > > When systems are under high load... > - will dtrace buffers be expanded to meet demand? I'm not 100% sure of the details regarding how buffers work in DTrace, but here's my understanding. Buffers are needed for data recording, but the size of the buffers required depends on what sorts of data recording actions the user is doing. If the user is printf()ing large amounts of packet data on a per-packet basis for example (I suspect this is the case you have in mind?) the principal buffer could indeed be filled before such data is displayed (with printf(), the issue is the interim between recording and reading out the data for display as I understand it). There are multiple buffering policies - "switch" (the default), "fill" and "ring". See http://wikis.sun.com/display/DTrace/Buffers+and+Buffering ..for more details on each. If the user expects buffer exhaustion will be an issue, they can set the buffer size explicitly with the bufsize option. > - will performance be slowed to ensure there is no loss with respect > to dtrace probes? As I understand it, the DTrace probes will always fire, but if buffers are full (in the default "switch" buffering policy case) data drops are reported as something like the following: dtrace: 11 drops on CPU 0 For the "fill" policy, tracing is halted when the buffer is full. > - is there any scope for being able to communicate that dtrace has > missed a packet (or two), should that happen if all of the buffers are > full? (This is only important for TCP) Data drops are counted on a per-CPU for the default "switch" buffering policy and the user is notified when such drops occur. One thing I should probably note is that while it's certainly possible to trace TCP data on a per-segment basis, I've found that I rarely end up using scripts that are heavy on data-recording actions like this in practice. Most of the use cases I've found involve asking questions about how TCP activity relates to other system abstractions (e.g. which processes or zones are sending a lot of TCP traffic?) or they involve aggregating data reasonably coarsely (e.g what is the average connection/first-byte latency by host? What is the mean round-trip time from transmission to ACK averaged per connection etc). In such cases, drops are rarely if ever encountered. Thanks! Alan > > Darren > From sebastien.roy@oracle.com Wed May 5 08:52:59 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o45Fqx4h028107 for ; Wed, 5 May 2010 08:52:59 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o45FqwbQ022957 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 5 May 2010 08:52:59 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1Y00C4HESA3X00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 05 May 2010 09:52:58 -0600 (MDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1Y0085BESA3630@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 05 May 2010 09:52:58 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o45Fqvq4026634 for ; Wed, 05 May 2010 15:52:57 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o44MCYKv012839 for ; Wed, 05 May 2010 15:52:55 +0000 (GMT) Received: from abhmt021.oracle.com by acsmt354.oracle.com with ESMTP id 239404931273074703; Wed, 05 May 2010 08:51:43 -0700 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 05 May 2010 08:51:41 -0700 Date: Wed, 05 May 2010 11:51:37 -0400 From: Sebastien Roy Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BDD46AB.8090107@oracle.com> To: Alan Maguire Cc: PSARC-ext Message-id: <4BE19409.4020408@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4BE19459.000B:SCFMA4539814,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100412 Lightning/1.0b1 Thunderbird/3.0.3 Status: RO Content-Length: 398 On 05/ 2/10 05:32 AM, Alan Maguire wrote: > As per the subject line, the updated spec is attached for > the above fasttrack. In the interim, the project team have > worked offline with Kacheong and Darren to address their > concerns. We have also incorporated SACK, PMTU and > congestion window info into the tcpsinfo_t at Nico's > suggestion. Thanks! This update looks good, +1 on the case. -Seb From ahl@eng.sun.com Wed May 5 11:04:55 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o45I4tml003325 for ; Wed, 5 May 2010 11:04:55 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o45I4mEn063965; Wed, 5 May 2010 12:04:53 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L1Y0041ZKW4JI00@nwk-avmta-2.sfbay.sun.com>; Wed, 05 May 2010 11:04:52 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L1Y00KYZKW2ZF90@nwk-avmta-2.sfbay.sun.com>; Wed, 05 May 2010 11:04:50 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o45I4oet001907; Wed, 05 May 2010 18:04:50 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o45HpZfO006782; Wed, 05 May 2010 18:04:48 +0000 (GMT) Received: from abhmt015.oracle.com by acsmt353.oracle.com with ESMTP id 217039511273082648; Wed, 05 May 2010 11:04:08 -0700 Received: from [192.168.3.209] (/198.144.208.55) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 05 May 2010 11:04:08 -0700 Date: Wed, 05 May 2010 11:04:01 -0700 From: Adam Leventhal Subject: Re: updated spec for DTrace TCP and UDP providers [PSARC/2010/106] In-reply-to: <4BE0D120.6010707@Sun.COM> To: Darren Reed Cc: Alan Maguire , PSARC-ext Message-id: <45500EF4-0B77-4511-9066-B00630F7DD56@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1078) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4BE1B341.017D,ss=1,fgs=0 References: <4BDD46AB.8090107@oracle.com> <4BE0D120.6010707@Sun.COM> Status: RO Content-Length: 956 > I do have one question though (but only indirectly about this case)... > > When systems are under high load... > - will dtrace buffers be expanded to meet demand? > - will performance be slowed to ensure there is no loss with respect to dtrace probes? > - is there any scope for being able to communicate that dtrace has missed a packet (or two), should that happen if all of the buffers are full? (This is only important for TCP) Hey Darren, These are obviously outside of the scope of this case. These questions are answered in the DTrace documentation and in the original DTrace PSARC material (2001/466). Briefly, DTrace buffers are not expanded on demand (indeed, that's architecturally impossible), no throttling takes place, and DTrace has mechanisms for expressing the fact that trace data has been dropped (a necessary consequence of the scope of DTrace). Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl