INCLUDE_DATA

Archive for the ‘Database’ Category

MySQL webservice storage engine

Thursday, July 16th, 2009

As you probably know, MySQL works by having a pluggable storage engine which handles the retrieval and storage of data for the mysql engine that sits on top of it. The nice thing about it is that there is an api and an example storage engine so you can write your own.

Within Yahoo!, there is an indexing product called Vespa, which is used to store structured, indexed data, which can be retrieved by webservice calls. Due to the problems with getting and aggregating data from disparate datasources, I thought it would be nice to create a storage engine which interfaces with this webservice to allow queries from within mysql for both local mysql tables and data held within this index. The following Jing movie is a demo of what the storage engine can currently do:

Storage Engine Demo

Stop Using select *

Wednesday, August 6th, 2008

In my time trawling through various systems’s DB code, I see we are using “select *” in many places. This has to stop, and we need to specify the columns we want explicitly every time. The reasons for this are:

- A change to the table structure can break the code
- We end up getting all the columns from the DB when we almost certainly don’t need to. Some of the columns can be enormous, and it is a waster of DB and network resource in getting this data.
- A change to the query can add many more columns to be returned.
- It can make a difference to the query plan the DB uses, and can be many orders of magnitude faster to specify columns individually.

Using “select *” is lazy, suboptimal and a coding time bomb.



Dropping a Repository the easy way

Wednesday, June 25th, 2008

In previous OEM posts I have been dropping the Repository in a variety of ways. In a moment of Instruction Reading, something I rarely do because I love to conform to stereotypes, I found this little obvious-when-you-know beauty:

$ORACLE_HOME/sysman/admin/emdrep/bin/RepManager

[oracle@bbmos01 ~]$ $ORACLE_HOME/sysman/admin/emdrep/bin/RepManager $HOST $PORT $SID -action drop
Enter SYS user’s password :
Enter repository user name : SYSMAN
Enter repository user password :
Getting temporary tablespace from database…
Found temporary tablespace: TEMP
Checking SYS Credentials … OK.
Dropping the repository..
Quiescing DB … Done.
Checking for Repos User … Exists.
Repos User exists..
Clearing EM Contexts … OK.
Dropping EM users …
Done.
Dropping Repos User … Done.
Dropping Roles/Synonymns/Tablespaces … Done.
Unquiescing DB … Done.
Dropped Repository Successfully.

Ho hum. It can do it all.

Downgrading an OEM repository from 10.2.0.3 to 10.1.0.5

Wednesday, May 21st, 2008

If you ever have the misfortune of downgrading an Oracle server, you can run into some silly problems. One such issue raised its head when I needed to downgrade a server for a different project. I downgraded a 10.2.0.3 server to 10.1.0.5 and had a 10.2.0.3 OEM repository already installed. I exported all the schemas and reimported once I had downgraded.

I started the DB console and everything seemed fine, but when browsing to the OEM, I logged on and got a screen showing that it had not connected to the instance. A quick look at the logs followed. You can see the following errors in the emagent.trc file:

2008-05-21 00:57:47 Thread-4120902576 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2008-05-21 00:57:47 Thread-4120902576 WARN upload: FxferSend: received http error in header from repository: http://oracle.oakdene.net:5500/em/upload/
ERROR-400|ORA-06550: line 1, column 18:
PLS-00302: component ‘OBTAIN_LOCK_FOR_EMD_URL’ must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored

I decided that I would accept not having the old repository data for now, I could always import into another schema and do a post fix data import. I needed to drop the repository and recreate the 10.1.0.5 version (theemca commands are different between 10.2 and 10.1).

[oracle@oracle ~]$ emca -x OAKDENE_SID

STARTED EMCA at Wed May 21 01:54:52 PDT 2008
May 21, 2008 1:54:52 AM oracle.sysman.emcp.EMConfig stopOMS
INFO: Stopping the DBConsole …
Enterprise Manager configuration is completed successfully
FINISHED EMCA at Wed May 21 01:54:57 PDT 2008

I then tried installing the db console. I used a response file called emca.txt. This didn’t work properly as can be seen below:

[oracle@oracle ~]$ emca -f emca.txt

STARTED EMCA at Wed May 21 01:55:24 PDT 2008
Password for dbsnmp:
Password for sysman:
Password for sys:
May 21, 2008 1:55:32 AM oracle.sysman.emcp.EMConfig checkConfiguration SEVERE: Repository already exists. Fix the error(s) and run EM Configuration Assistant again in standalone mode. Could not complete the configuration. Refer to the log file for details

I decided to drop the repository manually by dropping the SYSMAN user:

[oracle@oracle ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.1.0.5.0 - Production on Wed May 21 01:55:42 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> drop user sysman cascade;

User dropped.

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
[oracle@oracle ~]$ emca -f emca.txt

STARTED EMCA at Wed May 21 02:01:35 PDT 2008
Password for dbsnmp: Password for sysman: Password for sys:
May 21, 2008 2:01:43 AM oracle.sysman.emcp.EMConfig updateReposVars
INFO: Updating file /home/oracle/oracle10db/sysman/emdrep/config/repository.variables …
May 21, 2008 2:01:48 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Error creating the repository. Repository creation could not be completed successfully
May 21, 2008 2:01:48 AM oracle.sysman.emcp.EMConfig perform
INFO: Please refer to the log file at /home/oracle/oracle10db/cfgtoollogs/emca_repos_create<DATE>.log for more details.
Could not complete the configuration. Refer to the log file for details
[oracle@oracle ~]$ vi /home/oracle/oracle10db/cfgtoollogs/emca_repos_create2008-05-21_02-01-43-AM.log

But it still failed. In the log mentioned here is is quite obvious what has happened:

CREATE ROLE MGMT_USER
*
ERROR at line 1:
ORA-01921: role name ‘MGMT_USER’ conflicts with another user or role name

After fixing this issue (and others) by repeating, there are a series of other objects which need dropping as well:

[oracle@oracle ~]$ cat drop_sysman.sql

drop public synonym SETEMVIEWUSERCONTEXT;
drop role mgmt_user;
drop public synonym mgmt_target_blackouts;
drop user sysman cascade;
drop user mgmt_view cascade;
exit

[oracle@oracle ~]$ sqlplus ‘/ as sysdba’ @drop_sysman

SQL*Plus: Release 10.1.0.5.0 - Production on Wed May 21 02:02:16 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

Synonym dropped.

Role dropped.

Synonym dropped.

User dropped.

User dropped.

Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

And this works now, the repository gets create properly:

[oracle@oracle ~]$ emca -f emca.txt

STARTED EMCA at Wed May 21 02:24:32 PDT 2008
Password for dbsnmp: Password for sysman: Password for sys:
May 21, 2008 2:24:42 AM oracle.sysman.emcp.EMConfig updateReposVars
INFO: Updating file /home/oracle/oracle10db/sysman/emdrep/config/repository.variables …
May 21, 2008 2:26:18 AM oracle.sysman.emcp.EMConfig createRepository
INFO: Creating repository …
May 21, 2008 2:26:18 AM oracle.sysman.emcp.EMConfig perform
INFO: Repository was created successfully
May 21, 2008 2:26:19 AM oracle.sysman.emcp.EMConfig addPortEntries
INFO: Updating file /home/oracle/oracle10db/install/portlist.ini …
May 21, 2008 2:26:19 AM oracle.sysman.emcp.EMConfig updateEmdProps
INFO: Updating file /home/oracle/oracle10db/sysman/config/emd.properties …
May 21, 2008 2:26:19 AM oracle.sysman.emcp.EMConfig updateConfigFiles
INFO: targets.xml file is updated successfully
May 21, 2008 2:26:19 AM oracle.sysman.emcp.EMConfig updateEmomsProps
INFO: Updating file /home/oracle/oracle10db/sysman/config/emoms.properties …
May 21, 2008 2:26:19 AM oracle.sysman.emcp.EMConfig updateConfigFiles
INFO: emoms.properties file is updated successfully
May 21, 2008 2:26:20 AM oracle.sysman.emcp.EMConfig startOMS
INFO: Starting the DBConsole …
May 21, 2008 2:27:54 AM oracle.sysman.emcp.EMConfig perform
INFO: DBConsole is started successfully
May 21, 2008 2:27:54 AM oracle.sysman.emcp.EMConfig perform
INFO: >>>>>>>>>>> The Enterprise Manager URL is http://oracle.oakdene.net:5500/em <<<<<<<<<<<
Enterprise Manager configuration is completed successfully
FINISHED EMCA at Wed May 21 02:27:54 PDT 2008

Perseverance is the key :).

Installing PokerTracker on a Mac

Wednesday, March 12th, 2008

Download the pokertracker installer and install using the unsupported software installation for crossover. I want to use the postgres database server as the database as I have always hated Access and getting that to work with crossover has had mixed reviews according to some forum posts I have seen, although crossover lists various Access versions as supported.

To install postgres I used the instructions on the developer.apple site - http://developer.apple.com/internet/opensource/postgres.html

I need to install fink (http://www.finkproject.org/download/index.php) as well. I am using Leopard, and at the time of writing, there is no binary, so I need to get the source. After downloading I realised I had no c compiler installed! A handy error message told me I needed the developer tools:

....
Checking cc... not found.
ERROR: There is no C compiler on your system. Make sure that the Developer
Tools are installed.

Got the XCode3 stuff from http://developer.apple.com/tools/download/ and just installed the XCodeTools.mpkg package. This took just under 10 minutes to install, most of the time seemed to be installing documentation, which is nice. Seems a bit strange that a Mac doesn’t come with any compiler, but I am used to Linux when it comes to non-Windows machines, and this is my first Mac.

I have used WebObjects before and had no interest in that, so left it out. Reminds me too much of a sketchy project I worked on in another lifetime.

Once the dev tools are all installed, I can run the fink ./bootstrap command again. Unsurprisingly, I went for the defaults for most of the options, although I did turn on the unstable option as I like new broken stuff and I went for UK based mirrors…
I then was told to make a cup of tea in the INSTALL notes, so I did. At 23.58. Lets see how long this takes :)
… good job i waited as first curl call timed out :) happily downloading from another mirror.

I also notice, as I sit back down that my MacBook Pro is making more fan noise than ever. Hardest its worked so far methinks. At least it now knows what is ahead of it.

While this was grinding away, I downloaded the PostgreSQL source, as there is obviously no binary for OSX. I went for the 8.3 version and seeing as I saw bzip2 flash past on this fink thing, I went for the postgresql-8.3.0.tar.bz2 source files.

Fink finished at 00.11. Might be a late one, but I am warming to this task I think.

Now the instructions say install the readline using fink, but this happens:

me:fink-0.28.0 me$ sudo  /sw/bin/fink install readline
Password:
Scanning package description files..........
Information about 268 packages read in 0 seconds.
Failed: no package found for specification 'readline'!

This sucks. I added /sw/bin to my $PATH to make things easier and had a look at the other options for fink. A list was what i was after and this gave me:

me:fink-0.28.0 me$ fink list readline
Scanning package description files..........
Information about 268 packages read in 0 seconds.
 p   term-readline-pm588                      [virtual package]

This doesn’t look right. As readline is a GNU app, I wondered if the gnu mirror was cocked up, so went looking for a conf file rather than having to rerun the configuration again. It is /sw/etc/fink.conf and all the mirrors are listed there. As the terminal window does not have infinite window buffer size, the other mirror options had vanished into terminal nirvana, so I checked the gnu site for different mirrors (I could just download the readline source from here, but I am determined now). I added Mirror-gnu: ftp://ftp.cwru.edu to the list. This did sweet FA. I ran a fink selfupdate and watched the Dual Core go. I did have a lot of mirror trouble it seemed. Not sure if my connection is crapping out or if they are just unavailable. Got one in the end.

me:fink-0.28.0 me$ fink list readline
Information about 6504 packages read in 1 seconds.
     readline               4.3-1028          Comfortable terminal input library
     readline-java          0.8.0-1024        Readline support for Java
     readline-shlibs        4.3-1028          Comfortable terminal input library
     readline5              5.0-1005          Comfortable terminal input library
     readline5-shlibs       5.0-1005          Comfortable terminal input library
     term-readline-gnu-pm5  1.15-1004         Perl extension for the GNU Readline
     term-readline-pm       1.0203-12         Minimal interface to Readline
 p   term-readline-pm588                      [virtual package]

Cool. Went for the 5 version. More Core Duo action.

On to Postgres, files downloaded, we are nearly ready to crack on with this latest nonsense:

me:Downloads me$ md5 postgresql-8.3.0.tar.bz2
MD5 (postgresql-8.3.0.tar.bz2) = 53d6816eac7442f9bc8103439ebee22e
me:Downloads me$ cat postgresql-8.3.0.tar.bz2.md5
MD5 (postgresql-8.3.0.tar.bz2) = 53d6816eac7442f9bc8103439ebee22e

Rock n Roll. I can feel the tiredness of getting up early because of a timezone mistake and an evening boxing training taking its toll.
The INSTALL document has a short install instructions at the top. I will go through it and only put stuff here that deviated from the expected. I have a feeling old Core Duo is going to get another pounding.

First up, we need to do some extra includes for the ./configure, as mentioned in the apple.developer doc /configure –with-includes=/sw/include/ –with-libraries=/sw/lib
Second, I don’t have gmake, I have make so I used that instead.
Third, I forgot to sudo the make install, so got permission problems. If that is the only cock-up left in me tonight I will be very happy. Install completed pretty quick..
Some graphical UI stuff next according to apple.developer doc. I added the postgres user. When this was done, I could create a data directory for the db and then initialise it:

me:postgresql-8.3.0 me$ sudo sh
sh-3.2# mkdir /usr/local/pgsql/data
sh-3.2# chown postgres /usr/local/pgsql/data/
sh-3.2# su -l postgres
me:~ postgres$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default database encoding has accordingly been set to SQL_ASCII.
The default text search configuration will be set to "english".

fixing permissions on existing directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers/max_fsm_pages ... 2400kB/20000
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

    /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
or
    /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

Oh man we are getting close now!

me:postgresql-8.3.0 me$ sudo su - postgres
me:~ postgres$ echo $PATH
/bin:/usr/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/pgsql/bin:/sw/bin
me:~ postgres$ pg_ctl -D /usr/local/pgsql/data -l logfile start
server starting
me:~ postgres$ createdb pokertracker
me:~ postgres$ psql pokertracker
Welcome to psql 8.3.0, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

pokertracker-# \q
me:~ postgres$

Started up the PokerTracker application through crossover and lo and behold, it fails with:

unable to connect to the PTREG2 database

Goddamnit, is it trying to connect to Access DB first. If I remember correctly I think it does, and you have to convert the DB to be a Postgres one. Shite. It is now 1.30am and I need sleep. Next post is how I got hold of a version of Access 2000 :).