python len() for record offsets

 From a colleague:

In case you ever have to do an f.seek():

When calculating record offsets, python only does a consistent and correct len() value if your file is opened in binary.

Performance illusion from using repetitive test data

We often copy a few fake records to create a large input file with which to test how the code handles it.

It's a useful test; but don't use it to gauge performance. 

Data from repeated records can be cached and reused. If your production data will vary, then it won't get the same use out of caching. Caching could even hurt performance.

Windows batch script - decimal points and comma separators

Adding a decimal place to express ms as seconds:

C] echo %ms:~0,-2%.%ms:~-2,2%
123.45

C] echo %m2:~0,-2%.%m2:~-2,2%
12.34

C] set m1=123

C] echo %m1:~0,-2%.%m1:~-2,2%
1.23

C] set m0=23

C] echo %m0:~0,-2%.%m0:~-2,2%
.23

C] set m=2

C] echo %m:~0,-2%.%m:~-2,2%
.2

C] set m=02

C] echo %m:~0,-2%.%m:~-2,2%
.02

To insert commas, not so simple:

C]  set m=123456789

C]  echo %m%
123456789

C] echo %m:~-3,3%
789

C] echo %m:~-6,3%,%m:~-3,3%
456,789

C]  echo %m:~-9,3%,%m:~-6,3%,%m:~-3,3%
123,456,789

C]  echo %m:~-12,3%,%m:~-9,3%,%m:~-6,3%,%m:~-3,3%
123,123,456,789


More complex solutions at https://stackoverflow.com/questions/28700043/thousands-separator-of-an-integer-value


C]  echo %m%
123456789

C]  if %m% GTR 99999999 (set c=%m:~-9,3%,%m:~-6,3%,%m:~-3,3%)C]  echo %c%
123,456,789

C]  set n=12345

C]  if %n%  GTR 99999999 (set c=%n:~-9,3%,%n:~-6,3%,%n:~-3,3%) else ( if %n% GTR 9999 (set c=%n:~-6,3%,%n:~-3,3%) )

C]  echo %c%
123,345       WRONG!!

Using RegExp in Notepad++ to convert all RollingFileAppender to DailyRollingFileAppender in a log4j.properties file

1) Change RollingFileAppender to DailyRollingFileAppender

Search for

log4j.RollingFileAppender

Replace with

log4j.DailyRollingFileAppender


2) To comment out the MaxFileSize line:

Search for

log4j\.appender\.[a-zA-Z]+\.MaxFileSize

Replace with

#$&

This works because $& matches the entire expression matched.
(Commenting out instead of deleting in case you want to go back to Rolling.)

3) To comment out the MaxBackupIndex line:

Search for

log4j\.appender\.[a-zA-Z]+\.MaxBackupIndex

Replace with

#$&


4) To insert the DatePattern='.'yyyy-MM-dd line just before existing log4j.appender.SOMETHING.layout= line:

Search for

(log4j\.appender\.[a-zA-Z]+\.)layout=

Replace with

${1}DatePattern='.'yyyy-MM-dd\n$&

This works because:
The parentheses marking from "log4j" to just before "layout" marks out the part returned by ${1}
Then the "\n" adds a newline
Then the $& restores the beginning of the layout= line.

 

-->

TrackMyUniverse

I’ve lately been looking at GitHub and listening to python podcasts trying to think of a python project to do. I remembered a Prolog project I wrote for a class I was taking at Harvard Extension back in the ‘80s. Prolog has a built-in inference engine, so I wrote a story chronology tool. E.g. if event A happens before event C and B before A then B was before C.

For decades, now, I’ve been writing bits of stories that all take place in the same space fantasy universe.
I have boxes and Google docs folders full of drafts.
There are some sort of finished parts: A few TV mini-series scripts, a full-length screenplay, a novella, and a novel that once could be read as finished until, under the influence of its 3rd or 4th writing group, I tore it apart and never pieced it back together again.

Whenever I return to writing about my universe I need to find the correct draft and update there.

Sometimes, I’d like to be able to open my computer and just start typing.

Maybe I could write a python tool for this.
It could use a noSQL database- since it would be literally collections of documents.
One point of access would be a character list.
Another would be places: each event or story fragment occurs at a particular place.
Events are characterized by characters and places.
Places and characters have event histories.
Characters are connected by family relationships from which obvious chronologies can be inferred. No significant time travel, so events when parents are young can be inferred to occur before events when their children are adults, etc.
Characters are also connected by time spent together- in the same place or in the same spaceship.

I’d also like to tie in version control for saving multitudinous versions of paragraphs, chapters, scenes, lines of dialogue, whatever.

Eventually, one could walk into this universe and look around.

Windows - using %CD% for current directory and levels

For windows scripting, you can access levels above %CD%. Just remember to use backslash, not forward slash:

C:\Users\mharrison>echo %CD%
C:\Users\mharrison
C:\Users\mharrison>dir %CD%\..\..
 Volume in drive C is Windows
 Volume Serial Number is E431-FA9F
 Directory of C:\
03/03/2017  11:02 AM    <DIR>          DRIVERS
04/11/2018  07:38 PM    <DIR>          PerfLogs
05/31/2019  02:25 PM    <DIR>          Program Files
05/15/2019  09:04 AM    <DIR>          Program Files (x86)
03/25/2019  05:33 PM    <DIR>          test
05/13/2019  10:56 AM    <DIR>          Tools
04/09/2019  09:21 AM    <DIR>          Users
08/08/2019  06:02 PM    <DIR>          Windows
               0 File(s)              0 bytes
               8 Dir(s)  61,991,501,824 bytes free

C:\Users\mharrison>dir %CD%/..
Invalid switch - "..".

Also, check out this great tool for formatting code for blogging on blogger:
http://formatmysourcecode.blogspot.com/

Centos Firstboot - No Way Out

So I continued following the advice in CentOS 6 Linux Server Cookbook.
I installed firstboot.
    yum -y install firstboot
    chkconfig firstboot on
One is supposed to be able to navigate using arrow keys.
That worked until this screen:

But suddenly, the tab keys worked! Now I can leave.
Never mind.

Trying to set up new CentOS 6.5 from Skytap Template

OK, figured how to log in from Settings / Credentials - Default password comes with the template.
Using CentOS 6 Linux Server Cookbook from www.safaribooksonline.com.

1st I used the directions from https://help.skytap.com/vmware-tools-linux.html to install VMWare tools.
I had to do a "mkdir /media/cdrom" since /media was there, but not /media/cdrom.
After that, the mount command and the rest of the instructions worked.

Then, following the Cookbook, I did

yum -y update

This took a while. It looked like a lot of packages were updating.

Then I got the error message:

kernel panic not able to mount root fs on unknown block(0,0)

I tried rebooting and got a message about operation failed because VMWare Tools are still loading:

There are other options:

I don't have much to lose, so I try Reset. But I get the same message about VMWare Tools still loading. So I click the Power off button...

It is now Powered Off.

I click the Run button... 

I open the VM:

Hmm.

Well, I can delete the VM and start again.

Or give up on creating my own, and just copy a more set-up VM from a co-worker.

Oh well.

5 minutes later:  OK, but, WhatifItry:

1) Create a new VM from Template

2) Don't add CPU, disk space, memory, like I did before

3) Do the yum -y update BEFORE trying to install the VMWare tools.

---

Here we go again:

...

It rebooted, and let me log in.

Should I install VMWare tools now?

python: Read in generic lookup table from tab-separated text file with field-names on top line

not beautiful yet, but here's a start:

lookup={}

fd=open("tab_separated_fields.txt", 'r')

try:

    headers=fd.readline()

    names=headers.split()

    num_fields=len(names)

    print "%d fields in %s" % (num_fields,names)

    for row, line in enumerate(fd,1) :

        print "row %d: line=%s" % (row,line)

        if len(line) > 10:

            values=line.split('\t')

            if len(values) == num_fields:

                lookup[str(row)] = {names[i]:(values[i]).strip() for i in range(num_fields)}

            else:

                print "row %d doesn't have %d values" % (row,num_fields)

finally:

    fd.close()

print lookup

Next, see if csv reader makes it better.
Actually, csv.DictReader.
e.g https://github.com/pargery/csv_utils2/blob/master/SimpleCSVReporter.py

Mediocre Windows Batch Script to Measure Time

Well, it uses clock time, and would be more useful as a called subroutine, but this is what I've got so far:

rem Test Timing

rem You might need to set this in later versions and also use !time 
setlocal enabledelayedexpansion

set start_time=%time%
echo start_time is %start_time%

pause
rem Replace above 'pause' with activity to be timed (approx).

set end_time=%time%

rem Now calculate time difference

echo started %start_time%
echo ended %end_time%


rem To get a substring, do this:
rem    %variable:~num_chars_to_skip,num_chars_to_keep%

rem Here we also prepend "1" to 2-digits and then subtract 100 
rem   to avoid numbers like "02"
rem   which can cause errors in the windows batch interpreter.

set sh=1%start_time:~0,2%
set eh=1%end_time:~0,2%
set sm=1%start_time:~3,2%
set em=1%end_time:~3,2%
set ss=1%start_time:~6,2%
set es=1%end_time:~6,2%
set sms=1%start_time:~9,2%
set ems=1%end_time:~9,2%

echo sh %sh%, eh %eh%, sm %sm%, em %em% and sms %sms%, ems %ems%
pause

set /A start_secs=(%sh%-100)*3600 + (%sm%-100)*60 + (%ss%-100)
set /A end_secs=(%eh%-100)*3600 + (%em%-100)*60 + (%es% - 100)

set /A start_ms=(%start_secs%)*100 + (%sms% - 100)
set /A end_ms=(%end_secs%)*100 + (%ems% - 100)

set /A diff=%end_ms% - %start_ms%

echo %start_ms% to %end_ms% is %diff% milliseconds 
pause

Lots of "pause"s to check how it's going.
You can take them out.
Better versions welcome, too.