

Supporting benchmarking data
---
We need random data on the client side for creating TraceIDs. We need to see how fast we can get it. If the random source runs out of data, then our client will pause while more entropy is created.

### Random data

	time cat /dev/urandom | head -c 1000000000 > /dev/null
	1:30

11.11M/sec random bytes

###app server performance

4000-8000 requests per second
40 app servers
100-200 requests/sec per app server


Ideas for visualization
---
http://www.graphviz.org/content/how-display-edges-when-you-click-node
http://code.google.com/p/svgpan/
http://ushiroad.com/jsviz/about/
http://www.graphviz.org/content/fsm
http://grokbase.com/t/gg/golang-nuts/12cd8qv3q2/go-nuts-visualising-import-graphs
https://github.com/davecheney/graphpkg/blob/master/main.go

setup
---
sudo ln -s  /mnt/media/cbbde44f-731f-4202-b2a8-b9d0d2aa302a/ /var/lib/trace

collects listens on :11142
haproxy listens on 127.0.0.1:12310

supervise
---

/etc/service/collect/:
down  log  run	supervise

/etc/service/collect/log:
run  supervise

/etc/service/collect/log/supervise:
control  lock  ok  status

/etc/service/collect/supervise:
control  lock  ok  status

cat /etc/service/collect/run 
	#!/bin/bash

	set -e

	exec 2>&1 
	exec setuidgid jtv /home/users/martin.hess/collect_linux_amd64 -laddress=":11142" -dbdir="/var/lib/trace"

cat /etc/service/collect/log/run 
	#!/bin/bash

	exec logger -p local3.info -t collect



sudo mv collect /etc/service
sudo svstat /etc/service/collect/
sudo svstat /etc/service/collect/log/
sudo svc -o /etc/service/collect/
sudo svc -d /etc/service/collect/
sudo svc -u /etc/service/collect/
tail -F /var/log/jtv/collect.log

How to connect to Trace:Collect
---
If HAProxy Backend is configured correctly than you connect to it at:

	127.0.0.1:12310

You can verify that HAProxy is configured correctly by:

	$ grep tal1 /etc/haproxy/backend.conf 

    server tal1.sfo01.justin.tv-11142 tal1.sfo01.justin.tv:11142  check inter 5000

If it is not configured, you can connect directly at:

	tal1.sfo01.justin.tv:11142

The trace is stored at:

	tal1.sfo01.justin.tv:/var/lib/trace/


puppet
===
* config nginx
* test
	sudo puppet agent --test --tags="tal" --environment="tal" --noop
* run locally
	sudo puppet agent --test --tags="tal" --environment="tal" 
* nginx config file location
	/usr/local/nginx/conf/servers/tal.conf
* validate nginx config 
	sudo /etc/init.d/nginx configtest
* reload nginx config
	sudo /etc/init.d/nginx reload
* look at the nginx log
	tail /var/log/nginx/tal_error.log

[puppet]> git --no-pager diff   master..tal
diff --git a/manifests/classes/tal.pp b/manifests/classes/tal.pp
new file mode 100644
index 0000000..2f08d9e
--- /dev/null
+++ b/manifests/classes/tal.pp
@@ -0,0 +1,28 @@
+class tal{
+
+	include software::http
+
+	file { "/var/lib/trace_reports": 
+		ensure => directory,
+		mode => 0777,
+		owner => root,
+		group => root,
+	}
+
+	$svc_name = tal
+	
+    file { "${svc_name}_nginx_conf":
+        ensure       => present,
+        path         => "/usr/local/nginx/conf/servers/${svc_name}.conf",
+        sourceselect => first,
+        require      => Package[nginx],
+        owner        => 'root',
+        group        => 'root',
+        source       => [
+            "puppet:///modules/files/conf/nginx/${svc_name}/${cluster}.conf",
+            "puppet:///modules/files/conf/nginx/${svc_name}/${::hostname}.conf",
+            "puppet:///modules/files/conf/nginx/${svc_name}/default.conf"
+        ],
+    }	
+
+}
\ No newline at end of file
diff --git a/manifests/site.pp b/manifests/site.pp
index 3f966e5..a25a848 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -444,3 +444,11 @@ node /^buildslave\d+/ {
     class { base: }
     class { "${cluster}": }
 }
+
+node /^tal\d+/ {
+    $cluster = "tal"
+    class { base: }
+    class { "$cluster": }
+}
+
+
diff --git a/modules/files/files/conf/nginx/tal/default.conf b/modules/files/files/conf/nginx/tal/default.conf
new file mode 100644
index 0000000..afb52ae
--- /dev/null
+++ b/modules/files/files/conf/nginx/tal/default.conf
@@ -0,0 +1,13 @@
+    server {
+      listen  80;
+      server_name localhost;
+      root /var/lib/trace_reports;
+
+      access_log /var/log/nginx/tal_access.log;
+      error_log /var/log/nginx/tal_error.log;
+
+     location / {
+            autoindex  on;
+          }
+
+     }
[puppet]> 


Setup ElasticSearch and LogStash
===
sudo apt-get install openjdk-7-jre-headless -y
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.2.deb
sudo dpkg -i elasticsearch-1.0.2.deb

Adding system user `elasticsearch' (UID 107) ...
Adding new user `elasticsearch' (UID 107) with group `elasticsearch' ...
Not creating home directory `/usr/share/elasticsearch'.
### NOT starting elasticsearch by default on bootup, please execute
 sudo update-rc.d elasticsearch defaults 95 10
### In order to start elasticsearch, execute
 sudo /etc/init.d/elasticsearch start

 Logstash config for tcp
 ===
	input {
	 tcp {
	        codec => "json_lines"
	        mode => "server"        
	        port => 52345
	 } 
	}

	output {
	  elasticsearch { host => localhost }
	#  stdout { codec => rubydebug }
	}

Testing ElasticSearch Performance
===

Control
---

server side
---
	nc -l 52525 | sudo tee /var/lib/elasticsearch/dump.log > /dev/null

client side
---
	cat 2014-02-27-13-54-45.190155 | nc tal2.sfo01.justin.tv 52525

Test
---

client
---
	cat 2014-02-27-13-54-45.190155 | nc tal1.sfo01.justin.tv 52345


Debugging traffic
===

	sudo ngrep -q -W byline -d bond0 host 199.9.252.197

	sudo tcpdump -i bond0 host 199.9.252.197 -XX -A