ScienceSim Montoring

The ScienceSim simulators are instrumented to collect detailed statistics. The statistics are not associated with any user or detailed activities but instead cover general operational parameters – bytes in, CPU time simulating physics, etc.

There are four classes of information collected:

• system configuration (amount of memory, number of processors, …);
• overall system performance (uptime, load average, memory allocated, …);
• header information on all UDP packets sent; and
• region statistic packets

There is no user specific information collected so, for instance, the number of avatars logged into a region is collected but the names or identity of the avatar is not.

The contents of the following commands is captured daily:

• cat /proc/version: operating system version
• /usr/bin/lshw: detailed information on motherboard components
• /sbin/sysctl -a: list of current kernel parameters
• cat /proc/cpuinfo: detailed processor information

This captures a detailed picture of the hardware and operating system configurations.

The contents of the following commands is captured every few minutes:

• cat /proc/uptime:
• cat /proc/loadavg:
• cat /proc/stat:
• cat /proc/meminfo:
• cat /proc/diskstats:
• cat /proc/net/netstat:

This captures the usage counters for the CPU, memory, disk and network. This gives general information on the usage of the system resources by the simulator.

OpenSim.exe has been patched to save information on the UPD packets that are sent to all of the clients connected to the simulator. The following fields are collected from each packet:

• time: the microsecond timestamp of when the packet was queued for sending;
• type: the type of the packet message. There are several hundred packets types including avatar movement, texture download and object update.
• circuit: this is a random number used by the server to keep track of each client connection. While not identifying the identity of the client, this allows grouping the packets into sessions.
• length: the byte length of the UDP packet body

Each region (256 meter square area) sends regular statistics to each user. OpenSim.exe is patched to also store these region statistics to a file. These are generated every few seconds and each contains:

• Time dilation
• Simulator frames per second
• Agent update count
• FrameTime: time spent calculating frame
• NetTime: time spent processing network queues
• PhysicsTime: time spent calculating physics
• AgentTime: time spent calculating agent actions
• ImageTime: time spent processing images
• ScriptTime: time spent executing scripts
• OtherTime: other time
• Objects: number of objects in the region
• ScriptedObjects: number of scripted objects in region
• Agents: number of agents/avatars in region
• ChildAgents: number of tracked agents in adjacent regions
• ActiveScripts: number of active scripts
• LSLIPS: script language (LSL) instructions per second
• InPPS: in packets per second
• OutPPS: out packets per second
• PendingDownloads:
• VirtualSize: region size
• ResidentSize: region size resident in memory
• PendingLocalUploads:
• UnackedBytes:

As an example of the information collected, on January 30, 2009, the ScienceSim Senate met in world. In the hour of their first in-world meeting, a total of 9 avatars logged in. The following graphs show some of the statistics from that period.

This graph shows general simulator statistics for the one region holding the meeting. The period shown is from 6am to 10am PST. One individual logged in at 7am and spent an hour doing some physics (the yellow line). This you can clearly see in the second quarter of the graph (between 1000 and 2000 on the X axis). Beginning at 8am (the middle of the graph) meeting attendees began logging in.

The brown line uses the right hand scale and shows the number of agents in the region. All the other lines use the left hand scale. The meeting begins at 8am and lasts until 9am when the agents log out and the region activity levels out.

With this and all following graphs, you can click on it to see a higher resolution version.

(Click to see a higher resolution version.)

This next graph is a detail of the period from 7am to 8am when the one user was logged in. You can see the physics calculations (yellow line).

Click to see a higher resolution version.

The next two graphs are from the packet header data. These cover the same four hour period as the first graph. This graph is the total number of bytes sent for the most common UPD packet type. The key lists the packet types. You can see that the most bytes is sending the textures to the newly logged in clients.

Click to see a higher resolution version.

This graph is the packet count by type. So, rather than counting the number of bytes as in the previous graph, this shows the number of packets of each type. Here you can see that updating the agent's state (avatar location, actions and appearance) is the most common packet type.

Click to see a higher resolution version.