Linking of docker containers and further issues with buildbo_MySQL

Docker

All docker containers now exist, and one of the only things left to do is get all the containers playing nice with one another.

MySQL-app

I set out to breakout mysql into its' own docker containerand made good progress, but before proceeding further with debugging some setup problems, I checked out if anyone was opposed to using another mysql docker container as a foundation for our own. There are hundreds of mysql docker containers out there so it seemed silly to dupliate work if unnecessary. Noone had objections, so I went ahead and picked out a mysql docker container to use. I choseorchardup/mysqlfrom the Docker Index because it was pretty barebones and for the nice additional features it add in the form of being able to set environment variables in the container at runtime to do things like setup your own usernames, passwords, databases, etc, etc.

After awhile of trying to modify the run scripts that the orchardup/mysql image uses to launch the mysql server, I decided to back down for the time being. I was attempting to use orchardup/mysql as a base for our own mysql-app, so that I could then have our app do the additional loading of statusdb and scheduelrdb schemas. This proved to be a pain, and so rather than fight it further, I went with the redundant option of having buildapi-app and buildbot-app each individually load the schemas they needed into the database, regardless of if the schema already existed. I am not happy with this as a permanent solution for this development setup, but it should work well for our initial setup.

This also means that vagrant will now simply just need to pull the orchardup/mysql image, run it, forward ports, and link it with the other container apps, making this the lightest setup.

I modified buildbot-app and buildapi-app to use the newly created environment variables for the mysql app when connecting and using the databases (they appear upon running the docker containers when linking).

Buildbot-app

When I went to test buildbot-app, I ran into a an exceptions.ValueError: Malformed url

(Buildbot)root@96fbd42254f3:/# /start_buildbot.sh
mysql: option '-h' requires an argument
cd master && buildbot start $PWD
Following twistd.log until startup finished..
/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/scripts/logwatcher.py:52: PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process.
env=os.environ,
2014-05-21 05:35:52+0000 [-] Log opened.
2014-05-21 05:35:52+0000 [-] twistd 9.0.0 (/usr/bin/python2.7 2.7.3) starting up.
2014-05-21 05:35:52+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-05-21 05:35:52+0000 [-] monkeypatch_twisted_cbLogin applied
2014-05-21 05:35:52+0000 [-] Creating BuildMaster — buildbot.version: 0.8.2-hg-f6d9311d9246-production-0.8
2014-05-21 05:35:52+0000 [-] loading configuration from /Buildbot/build-master/master.cfg
2014-05-21 05:35:52+0000 [-] unable to import dnotify, so Maildir will use polling instead
2014-05-21 05:35:52+0000 [-] JacuzziAllocator 44938192: created
2014-05-21 05:35:52+0000 [-] nextAWSSlave: start
2014-05-21 05:35:52+0000 [-] nextAWSSlave: start
2014-05-21 05:35:54+0000 [-] JacuzziAllocator 37763792: created
2014-05-21 05:35:54+0000 [-] nextAWSSlave: start
2014-05-21 05:35:54+0000 [-] nextAWSSlave: start
2014-05-21 05:35:59+0000 [-] finished loading config file
2014-05-21 05:36:01+0000 [-] BuildMaster listening on port tcp:9000
2014-05-21 05:36:01+0000 [-] configuration update started
2014-05-21 05:36:01+0000 [-] configuration update failed
2014-05-21 05:36:01+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 628, in loadTheConfigFile
d = self.loadConfig(f)
File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 933, in loadConfig
d.addCallback(lambda res:
File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 190, in addCallback
callbackKeywords=kw)
File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 181, in addCallbacks
self._runCallbacks()
— —
File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 323, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 934, in
self.loadConfig_Database(db_url, db_poll_interval))
File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 1055, in loadConfig_Database
db_spec = DBSpec.from_url(db_url, self.basedir)
File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/db/dbspec.py", line 175, in from_url
raise ValueError("Malformed url")
exceptions.ValueError: Malformed url

The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.

make: *** [start] Error 1

It's possible this has to do with the mysql setup, as I possibly didn't link things up fully. More testing is necessary for tomorrow.

Buildapi-app

To run rabbitmq, mysql, and buildapi all linked together, run these commands in sequence

• docker run -d -p 5672:5672 -p 15672:15672 -p 4369:4369 -name rabbitmq rabbitmq-app
• docker run -d -p 3306:3306 -name=mysql orchardup/mysql
• docker run -t -i -p 8888:8888 -link rabbitmq:mq -link mysql:sql -name buildapi buildapi-app /bin/bash

This will drop you into a bash shell session in buildapi-app

When I attempt to run /start_selfserve_buildapi.sh I receive the following error:

root@e141d055c1c7:/# ./start_selfserve_buildapi.sh
Starting subprocess with file monitor
Running reloading file monitor
2014-05-21 06:37:13,352 Kombu connection revived
2014-05-21 06:37:13,353 Connected to amqp://selfserveagent@172.17.0.2:5672//
Traceback (most recent call last):
File "/usr/local/bin/paster", line 9, in
load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 84, in run
invoke(command, command_name, options, args[1:])
File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 123, in invoke
exit_code = runner.run(args)
File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 218, in run
result = self.command()
File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 276, in command
relative_to=base, global_conf=vars)
File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 313, in loadapp
**kw)
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 56, in fix_call
val = callable(*args, **kw)
File "/buildapi/buildapi/config/middleware.py", line 55, in make_app
config = load_environment(global_conf, app_conf)
File "/buildapi/buildapi/config/environment.py", line 66, in load_environment
init_scheduler_model(scheduler_engine)
File "/buildapi/buildapi/model/__init__.py", line 7, in init_scheduler_model
scheduler_db_meta.reflect(bind=engine)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 2342, in reflect
conn = bind.contextual_connect()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2284, in contextual_connect
self.pool.connect(),
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 209, in connect
return _ConnectionFairy(self).checkout()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 370, in __init__
rec = self._connection_record = pool._do_get()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 696, in _do_get
con = self._create_connection()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 174, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 255, in __init__
self.connection = self.__connect()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 315, in __connect
connection = self.__pool._creator()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 275, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (OperationalError) (2005, "Unknown MySQL server host 'SQL_PORT_3306_TCP_ADDR' (0)") None None

Again, this looks like a linking issue and more testing is necessary.

What's next?

The next steps are these:

• Resolve exceptions.ValueError in buildbot-app
• Resolve sqlalchemy.exc.OperationalError in buildapi-app
• Link rabbitmq, mysql, and buildapi and test that everything works
• Link mysql, and buildbot and test that everything works
• Link rabbitmq, mysql, buildapi AND buildbot and test that the whole package works
• See if there is a good way to load statusdb and schedulerdb schemas into mysql in a mysql-app setup built on the orchardup/mysql image. This would prevent the redundanc of loading schemas in buildapi-app and buildbot-app

Things I found useful here

• docker logs
• vboxmanage modifyvm boot2docker-vm –-nic1 delete http
• vboxmanage modifyvm boot2docker-vm –natpf1 "http,tcp,127.0.0.1,8888,,8888"

Things to look into

• VOLUME docker command
• Renaming apps as mozilla/buildbot-dev or mozilla/buildbot-dev
• Setting multiple natpf's for boot2docker testing