Subject: [LON-CAPA-dev] [Fwd: [Osdl_associates] OSDL's February, 2003 Newsletter]

From: Gerd Kortemeyer <[email-masked]>
Date: Fri, 14 Feb 2003 16:57:15 -0500

FYI below. The LON-CAPA project is an Associate of OSDL, and at some
point we might want to make use of their facilities and expertise.

Subject: [Osdl_associates] OSDL's February, 2003 Newsletter
From: OSDL News <[email-masked]>
To: [email-masked]
Date: 14 Feb 2003 13:21:09 -0800
List-Id: OSDL Associates Distribution List <osdl_associates.lists.osdl.org>

OSDL February, 2003 Newsletter

Table of Contents

1. OSDL Launches the Linux Stabilization Project
2. OSDL Database Test 2 Now Available in STP
3. OSDL-DBT-3 Has Been Released
4. High Level STP and PLM Reporting
5. News from the Carrier Grade Linux Work Group
6. Data Center Linux Update
7. Words of Witham

---

1. OSDL Launches the Linux Stabilization Project

Craig Thomas ([email-masked])
Test Engineering Manager

OSDL has initiated a project with the Linux community to aid in the 
transition of the 2.5 (development) kernel to the 2.6 (stable) version. 
The Linux Stabilization project provides the Linux developer community 
with quick feedback on each revision of the kernel by identifying 
performance issues and possible defects.  The activities involve 
executing various tests and posting the results to a single location.

The project is a collaborative effort within the open source community.

The current test activities outlined for this project are based on 
the intended targets of a Linux kernel:

- Servers (web, mail, LDAP)
- Enterprise class database usage
- Data Center Linux feature sets
- Carrier Grade Linux feature sets
- Desktop systems

In an effort to cover a broad range of kernel features, tests will be 
run on various hardware configurations within our lab for each new 
kernel revision.  The types of tests executed will be:

- correctness tests (functional, conformance, etc)
- stability tests (database simulations, file system 
  load, memory,  etc.)
- performance tests (network, scalability, I/O, etc)
- compilation tests (compiling on different architectures)
- binary compatibility tests

OSDL will provide the results of these test activities as additional 
information on the kernel's current state.

The results page is updated as new information becomes available.  

The the project results can be found at:

http://www.osdl.org/projects/26lnxstblztn/results/

We at OSDL would encourage you to send us results of your 
stabilization activities.  Join the Linux Stability mailing list at:

http://www.osdl.org/mailman/listinfo/linstab

---

2. OSDL Database Test 2 now available in STP

Mark Wong ([email-masked])
Member of the Technical Staff

OSDL Database Test 2 (OSDL-DBT-2), an on-line transaction processing 
(OLTP) workload, has been added to the Scalable Test Platform (STP).  
This is the second installment in our series of real-world database 
workloads in which kernel developers can test their patches to the 
Linux operating system. Request the 'dbt2-1tier' test on STP to 
evaluate your custom kernels.

OSDL-DBT-2 simulates a workload that represents a wholesale parts 
supplier operating out of a number of warehouses and their associated 
sales districts.  More information on OSDL-DBT-2 can be found at:

http://www.osdl.org/projects/performance/dbt2-page.html

Consistency results on Linux 2.4 and 2.5 are available on the project 
results page at:

http://www.osdl.org/projects/dbt2dev/results/

Currently, OSDL-DBT-2 operates with the SAP DB database but we are 
looking for interested volunteers to help port the test kit to other 
databases.  Please contact us if you are interested in contributing 
to the effort. 

---

3. OSDL-DBT-3 Now Released

Jenny Zhang ([email-masked])
Member of the Technical Staff

OSDL has released a third test kit in our series of real-world database 
workloads.  OSDL-DBT-3 simulates a real-world scenario of decision 
support for ad hoc queries.  Developers can use this test kit to 
determine the impact of a new Linux kernel patch to the database 
performance on the operation system.

Though DBT-3 derives from TPC-H, it provides more flexibility.  
For example, in addition to executing a complete TPC-H workload, 
DBT-3 provides the tester the ability to execute an individual test 
or an individual query.  The tester can also run user defined queries.  

DBT-3 provides the option to choose the database size used in 
the tests.  

The raw data size is defined by a 'scale factor'. For example, a scale
factor of 1 refers to a database with a raw data size of about 1GB. 
The actual database size varies depending on the database management 
system, and auxiliary data structures.  

At OSDL, we are currently running a DBT-3 workload against a 
database with a scale factor of 1.  We plan to scale up to a scale 
factor of 10. 
 
Information about DBT-3 is at:

http://www.osdl.org/projects/performance/dbt3-page.html

DBT-3 can be downloaded from the SourceForge project site:

http://sourceforge.net/projects/osdldbt

If you install and run the test, we would be very interested in 
hearing about your results.  You can post your results to the 
"OSDL Performance" mailing list:

http://lists.osdl.org/mailman/listinfo/osdl_performance

---

4. High Level STP and PLM Reporting

Bryce Harrington ([email-masked])
Senior Performance Engineer

Over the past months, STP and PLM have accumulated a wealth of data.  
Every patch uploaded into PLM gets automatically compile-tested a 
few different ways, and, of course, STP is continually running a 
variety of tests, both requested by our STP user base and the 
automated 'kautotest' system.  The one issue that folks have brought 
up time and again is that all this data is great, except how do you 
'get your arms around it'?  It's been hard to get a high level 
overview of all the results.

We have created a high level reporting tool to summarize both PLM 
and STP results for different kernel patches, indicating pass/fail 
for 26 different tests and 4 PLM filters.  (Pass is green, i
fail is red.)

The report can be found at:

http://www.osdl.org/stp_plm_report/

We have set it up to show 7 different patch series for both the 
2.4 and 2.5 kernels, along with some major patch sets for each.  
The tool is configurable so that people running STP 2.x outside 
OSDL can customize the data shown to be appropriate to their own 
testing series.  Further information on use and customization of 
the tool is available via its man page.  Watch for stp_plm_report 
in the STP 2.01 release.

---

5. News from the Carrier Grade Linux Work Group

Mika Kukkonen ([email-masked])
Carrier Grade Linux Roadmap Coordinator

We held a successful Face-to-Face meeting during LinuxWorld Expo 
in New York where we were able to establish clear roadmap for our 
version 2 specification work.  Our current goal is to finish and 
publish V2 specifications by the end of June, 2003.

One reason for our aggressive goal is to enable us to have a good 
idea which features we will want addressed the 2.7 tree of Linux 
kernel to address. We are planning to have a Face-to-Face meeting 
concurrent with Ottawa Linux Symposium, where the 2.7 tree is 
likely to be deeply discussed.

One area where we certainly would appreciate help is the security 
features in version 2. Security is a big and difficult issue, we 
know that there are people out there working on it already, and 
we have no wish to re-invent the wheel. Please chime in to our 
cgl_discussion mailing list if you hold Linux kernel security 
features near your heart.  

On Proof-of-Concept the biggest news is that Ericsson, one of our 
members in CGL-WG, has released as open source two important 
projects, namely TIPC (http://tipc.sf.net) and AEM 
(http://www.linux.ericsson.ca/aem). These will very likely be used 
as core building blocks of Carrier Grade Linux.

On validation we are focusing our efforts in POSIX testing. 
Our validation subgroup chair Rusty Lynch has been deeply involved 
in the 'Open POSIX Test Suite' project (http://posixtest.sf.net).

Another important area of focus for our validation efforts is IPv6, 
where there currently does not exist a good testing suite for Linux.  
We are planning to use tests made by TAHI-project 
(http://www.tahi.org) as a starting point.

In summary, things are looking quite busy and 2003 is starting to 
look like it is going to be a very interesting year. Once again, 
if you have any questions about CGL-WG or feel like you want to 
participate in our efforts, feel free to drop me ([email-masked]) 
an email. Cheers!

---

6. Data Center Linux Update

Stephen Hemminger ([email-masked])
Data Center Linux Roadmap Coordinator

Linux 2.7 requirements planning

The DCL technical board has received input from member and 
non-members about data center related requirements for Linux 2.7. 
This input is currently being organized and prioritized, but there 
is still time for more input.  

Please send the feature requests to the DCL roadmap coordinator 
<[email-masked]> by the end of February, for inclusion in the 
next round of requirements evaluation.

Linux 2.5 conduit

A Linux kernel source tree containing projects covered by the DCL 
requirements is maintained at OSDL.  This development conduit 
contains two pieces: a common subset for CGL and DCL, and a 
separate patch for DCL-only.

The most recent release is 2.5.59-dcl2. The patch set is broken 
into two pieces: a CGL/DCL common patch set containing:

Linux Trace Toolkit (LTT)        	Karim Yaghmour
Linux Kernel Crash Dump (LKCD)   	Matt Robinson, LKCD team
Kernel Probes (kprobes)          	Rusty Russell
Megaraid 2 driver                	Matt Domsch
DAC960 driver                    	Dave Olien

...and a DCL specific patch set containing:

Lost timer tick compensation   		John Stultz
Improved boot time TSC sync    		Jim Houston
Lockless gettimeofday          		Andi Kleen, 
					Stephen Hemminger
Perf monitor counters for x86  		Mikael Pettersson
RCU statistics                 		Dipankar Sarma
Scheduler tunables             		Robert Love

The latest release is available in downloadable patches from:

http://sourceforge.net/projects/osdldcl

...or the public BitKeeper repositories:

Common code:            bk://bk.osdl.org/linux-2.5-osdl
Common code + DCL:   	bk://bk.osdl.org/linux-2.5-dcl

---

7. Words of Witham

Timothy D. Witham
Lab Director

While there are lots of projects going on within the OSDL, the 
initial publication of the 2.5/2.6 kernel stabilization summary 
page stands out. It really isn't showing any data that didn't 
exist before. What it does is aggregate the reports that were 
already being run.   It reminded me that years ago when I used 
to teach new technical sales support people "Benchmarking 101" 
I would spend 1/3 of the class on how to present the data.  

While the goal of the testing that is performed at the OSDL is 
much different than a benchmark to support a sales situation, 
presentation of the results in a way that is quickly understood 
is just as important.  This isn't because you are trying to fool 
somebody with a fancy presentation; it is because people are busy 
and if they have to spend 1/2 hour trying to understand the 10,000 
lines of output they will put it off until they have time (which 
usually turns out to be sometime after the heat death of 
the universe.)

Oversimplification of the results also must be avoided.  A test 
that results in a single metric forces anyone (other than a manager) 
to evaluate and understand the meaning of that single number, 
again being put off until they have time.  

The really hard part is presenting data in a simple, easy to 
understand method that doesn't distort what the results really mean.  
A way that I judge if the results are presented in a fair and 
reasonable manner is to use what I learned from a great little 
book entitled "How to Lie with Statistics." The rule that I apply 
when trying to read someone else's data is that, anything used 
as an example in the book should be a flag that maybe things 
aren't as they seem.  And of course, I always avoid presenting 
my data using any of the bad forms mentioned in the book.
_______________________________________________
This message has been sent to registered OSDL Associates only.

To unsubscribe or change your options, visit:

http://lists.osdl.org/mailman/listinfo/osdl_associates