Linux: Getting a list of known hosts without all the key stuff


cut -d ' ' -f 1 ~/.ssh/known_hosts


Linux: In vi, to show line numbers do :set nu

:set nu

Turn off with

:set nonu

or

 :set nonumber

Other vi links so far:

• vi text wrap
• see tab and newline characters
• split screen (This doesn't always work.)

Python: Convert Python List to SQL "IN" List

    py_list=[123,2343,433,454]

    #convert the python list into a string to be used in SQL "IN" query
    sql_list=""
    quote = '\''
    for item in py_list:
        sql_list+= quote + item + quote

    query_string="SELECT ITEM FROM MY_TABLE WHERE MY_ID IN " + sql_list
    ...

Thank you, Janak.

Python IDE: PyScripter

Allen is using Eclipse.  Janak is using Stani's Python Editor.  I tried downloading Stani's some months back, but it never ran.  Janak says it needs configuring.

So I read A Beginner's Guide to IDE's, consulted http://wiki.python.org/moin/PythonEditors, and decided to go with the lightweight and "fast" single-language IDE PyScripter.

It downloaded and installed just fine.  I like that you can choose which version of Python to open it up with from the Open Program/PyScripter menu.  We're currently using 2.6, so I can look at our code in the IDE with 2.6 interpreting it.

Linux: uname -a tells you what version of Linux you're running

$ uname -a
Linux rhel54-64 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

There are other options to get selected info. You can do a "man uname" to see them.

python: iterating through csv.reader record

import csv

myReader=csv.reader(open(file_name,"rb"),delimiter='\t', quoting=csv.QUOTE_NONE) 

for record in myReader:
    print("len(record)="+str(len(record))
    for i in range(len(record)):
        print("record " + str(i) + "=" + record[i])

Though watch for Error Message

"not all arguments converted during string formatting"

python: printing more than one integer in a string

>>> a=5
>>> b=6
>>> c=7
>>>print("a=%d,b=%d,c=%d" % (a,b,c))

a=5,b=6,c=7

Linux: Testing for string equality in ksh

In .ksh this is how to do it:

$ plex="SIMPLEX"
$ if [ "$plex" = "SIMPLEX" ]
> then
> echo y
> fi
y
$ if [ "$plex" = "DU" ]
> then
> echo y
> else
> echo n
> fi
n

All of these do not work:

$ plex="SIMPLEX"
$ if (($plex=="SIMPLEX"))
> then
>   echo y
> else
>   echo n
> fi
y
$ if  (($plex=="DUP")) then echo t
> fi
t
$ if (($plex="SIMPLEX"))
> then
>   echo y
> else
>   echo n
> fi
n
$ if [["$plex"=="SIMPLEX"]]
> then
>   echo yes
> fi
ksh: [[SIMPLEX==SIMPLEX]]: not found
$ if [$plex="SIMPLEX"]
> then
>   echo yes
> fi
ksh: [SIMPLEX=SIMPLEX]: not found

Linux/Python: setting shell variable to value returned from python function

From stackoverflow.com.  This works with python 2.4:

$ f=$(python foo.py)
$ echo $f
FOOO

Where foo.py is:

def foo():
  print "FOOO"

if __name__ == '__main__':
   foo()

There was a recommendation to use sys.stdout.write("%s\n", fooPy()) instead of print(), but I got an error about sys.stdout.write() expecting one not two input variables when I tried it. If I use sys.stdout.write("%s\n" % foo()) or print(foo()) in the main clause then the output is:

$ python foo.py
FOOO
None
$ w=$(python foo.py)
$ echo $w
FOOO None

With

import sys
def foo():
  sys.stdout.write("FOOO")
if __name__ == '__main__':
   foo()

I get

$ python foo.py
FOOO$
$ w=$(python foo.py)
$ echo $w
FOOO
$

Linux: One minus the Line Count

To count the number of output records minus the header in one line of script:

CNT=$(($(wc -l file_with_header | awk '{print $1}')-1))

Linux: yesterday's date

This is cool:

$ date
Tue Jun 22 14:11:25 EDT 2010
$ date --date='1 day ago'
Mon Jun 21 14:11:36 EDT 2010

Linux: remove header and trailer lines from input file

• Use "sed '$d' infile > outfile" to remove the first line.
• Use "sed '1d' infile > outfile" to remove the last line.

$ cat hf.dat
head
one
two
three
foot
$ sed '$d' hf.dat > nofoot
$ cat nofoot
head
one
two
three
$ sed '1d' nofoot > noheadfoot
$ cat noheadfoot
one
two
three
$

python: take the average of a column of numbers from a file

Could use smoothing out, but easier than getting out the calculator:

def average(infile):
  acount=0
  asum=0
  f=open(infile,"r")
  g=open("avg_"+infile,"w")
  for line in f:
      num=float(line.strip())
      asum= asum + num
      acount=acount+1
      g.write('count=%d,' % acount)
      g.write('num=%f' %  num )
      g.write(',sum=%f \n' % asum)
  if acount > 0:
     average=asum/acount
     g.write('\n sum=%f \n' % asum )
     g.write('average=%f\n' % average)
  f.close();
  g.close();

def main(infile):
    average(infile)

if __name__ == "__main__":
    import sys
    main(*sys.argv[1:])

Note that float() of blank returns zero, so make sure there's no blank line at the end of your input file. Or, improve the script to check for that sort of thing.

>>> float()
0.0
>>> float( )
0.0
>>>

Linux: ^s freeze ^q unfreeze

If you accidentally hit Control-S and freeze your display, hit Control-Q to un-freeze it.

Linux: To see tab and newline characters in vi

From http://www.washington.edu/computing/unix/viqr.html

:set list

To stop seeing them:

:set nolist

And if you need to add in a tab, type control-v first and then the tab.

(To grep for tab)

Linux: ksh condition syntax, e.g. checking result of line count

ksh scripting can be simple to read, but hard to remember the syntax to write it:

# Number of lines in file
    LC=$(wc -l  $DIRNAME/$FILENAME | awk '{print $1}')
    if (("$LC" > 0))
    then
       nonempty file steps
    else
       empty file steps
    fi

• The value looked at has to be in quotes.
• You need to start the "then" on a new line, or have a semicolon before it.
  

  if [ ... ]; then
  

• That's two round parentheses surrounding the condition, because it's numbers not strings comparison.  
• Square brackets are for strings, checking file existence, or value returned from last function

if [[ -f "$DIRNAME/${FILENAME}.txt" ]]
then
...

OK, so I'm finding references to

if (test -d $1)

being the same as

if [-d $1]

and to the use of "-eq" for comparing integers, as in

if [$# -eq 3]

And if you use double square brackets, you avoid triggering an error if your variable is unset.

Good enough for now...

python: converting from written out date to numeric

Use the chart at the bottom of http://docs.python.org/library/datetime.html for the %B, %d, %m etc.

>>> import time
>>> f=time.strptime("November 29, 2010", "%B %d, %Y")
>>> f[0]
2010
>>> f[1]
11
>>> f[2]
29
>>> g=time.strftime("%m/%d/%y",f)
>>> g
'11/29/10'
>>> h=time.strftime("%m/%d/%y",time.strptime("October 4, 2012", "%B %d, %Y"))
>>> h
'10/04/12'
>>> time.strftime("%m/%d/%y",time.strptime("May 3, 2011", "%B %d, %Y"))
'05/03/11'
>>>time.strftime("%B %d, %Y",time.strptime("23-Jun-14","%d-%b-%y"))
'June 23, 2014'

py-sig Manchester NH: Goffstown Man with T-Shirt from PyCon reports:

~1100 people, the largest pycon every. 10% female. A thing of note.
Tracks, birds of a feather, lightning talks.
Takeaway messages:
1) Use distribute and pip rather than setup tools.Not easyinstall.
Peripheral Interchange Program.
Distribute keeps track of what's installed so it can uninstall too.
google for and download distribute_setup.py
then say easyinstall pip.
this will keep track of all the new stuff.
2) interesting session on greenlets, a lightweight threaded model..
3) David Beasley's talk on global interpreter was impressive.
charts on contention and giving up locks but taking them back up before something else could get them.
jpython doesn't have this problem.
unix had global kernel locks.
[discussion about tuning each lock manually. has to do with how interpreter works. deadlocks created. better to be slow than stopped.]
"If you're worried about the global interpreter lock, use Jython."
He was wearing a t-shirt saying "python is for girls"
4) No-sql databases
key-value
distribute copies across multiple servers
python bindings for these, not written in python since C is faster.
5) A nice presentation on testing, unit testing.
6) Ark was trying to get people together into a sprint to convert the rest of the libraries to python3.

Videos of PyCon sessions are on pycon.blip.tv

py-sig notes#3 how to teach intro to python

Start with "variables are not places where values are stored".
Then lists, dictionaries.
Then take apart some samples, probably some things in the cookbook.
Here's how you'd use the cookbook sample in a real program.
People who program are good at looking at what works and picking up how the language helped.
taking advantage of some feature of python.
part the language and part the environment.
Discuss portability.
In contrast, for example, with C, which really needs properly-done typecasting, to avoid issues such as little vs big endian.
vs more strongly typed languages like python

py-sig Manchester NH 7:20 and on, random notes

No formal presentation tonight. They're just giving newbie advice...

Link to mailing list: pysig.org or python.org, follow link to community, then mailing lists then "tutor" for beginning questions.  Kent Johnson is one of the regulars for "tutor" and is often at our meetings.

windows/linux portability issues:
single new line vs carriage return and newline.

#!path to interpreter
vs
on windows where the newline will cause the line to be ignored.

dos2unix to get rid of extra newlines.

watch out for tabs, so that they're not 4 or 8 spaces, since # of spaces matter
vs
just use 4 spaces.  convention?
4 per level (though some people use 2)

misc recommended editors:
sciTE
emacs
vim

if you want to do graphics, there's
pil=python imaging library

(you'll dance to anything by...)

The Greater NH Linux Users Group is a superset of this group.

At py-sig Manchester NH 7pm

To install py3 on fedora, type:

$sudo yum install python3

 or if you're already super user, just type

$yum install python3

or

$yum search python3 --enablerepo=rawhide

Linux: uncompress tar.Z when your Linux doesn't have uncompress

The second time I've had to look this up, so posting it here:

zcat filename.tar.Z | tar xvf -
or
gzcat filename.tar.gz | tar xvf -

vi text wrap

from http://jjinux.blogspot.com/2007/02/vim-soft-wrap.html

:set wrap linebreak textwidth=0

Useful when you're trying to view or edit a file with no page breaks...

Linux: We've got python2.4, we need python2.6 now, too.

pointers to YUM or RPM
If we had YUM, it would be easy:

yum upgrade python
yum install tkinter

RPM sounds scary.

And we also need to be able to still run python2.4 for some projects.

From  http://python.org/download/
Click on Python 2.6.4 compressed source tarball (for Linux, Unix or OS X)

Copy Python-2.6.4.tgz to where you want it.

tar xvzv  Python-2.6.4.tgz

produces a Python-2.6.4/ directory.

Here's the README.  My only question is what is the top-level directory to which they refer.  Is it where I want the final python2.6.4 lib to be?

...To start building right away (on UNIX): type "./configure" in the
current directory and when it finishes, type "make".  This creates an
executable "./python"; to install in /usr/local, first do "su root"
and then "make install".

then

Build instructions
==================
Before you can build Python, you must first configure it.
Fortunately, the configuration and build process has been automated
for Unix and Linux installations, so all you usually have to do is
type a few commands and sit back.  There are some platforms where
things are not quite as smooth; see the platform specific notes below.

...

Start by running the script "./configure", which determines your
system configuration and creates the Makefile.  (It takes a minute or
two -- please be patient!)  You may want to pass options to the
configure script -- see the section below on configuration options and
variables.  When it's done, you are ready to run make.

To build Python, you normally type "make" in the toplevel directory.
If you have changed the configuration, the Makefile may have to be
rebuilt.  In this case you may have to run make again to correctly
build your desired target.  The interpreter executable is built in the
top level directory.

Once you have built a Python interpreter, see the subsections below on
testing and installation.  If you run into trouble, see the next
section.

...

Ah, phew. I don't have to do this. We have a tar file on our company intranet with the approved 2.6 python libs already tarred up for the taking. If only it had been easier to find an hour ago when I looked... Worth finally asking the humans though. No matter how harried they look.

I leave this post anyway, in case someone reading this actually does have to install python.

Linux: setting the system time & date

I've had to manually reset the date on our Linux VM every day:

su -c "date MMDDHHMMYYYY"

The weird thing is the order: Month Day Hour Minute, then year.

e.g., today at 9:55am

su -c "date 031809552010"

I seem to recall when we first noticed that time was passing slowly on the VM that there were some possibly better solutions suggested by VMWare. I just haven't the time to work on them.

(Update May 2013:) The lazy way to collect the correct string to use is to log into another Linux box and type 

 date +%m%d%H%M%Y

That will return the string "052416532013" which you can copy and paste into the command on the server that needs its time updated.

$ su -c "date 052416532013"
Password:
Fri May 24 16:53:00 EDT 2013

Linux: diff -rq

To compare 2 directory structures to see if their files are the same or different.

Linux: grep for tab character

Two ways:

1. grep quote control-v tab quote
2. cat -T then pipe to grep '\^I'

First I created a file with tabs for testing.  I typed "Hi there" then the tab character, then return, then "this is a " then control-V, then the tab character then "real tab", and then the next line without tabs:

$ cat > foo
Hi there
this is a ^     real tab
isn't it.
$ cat foo
Hi there
this is a       real tab
isn't it.

1. You can grep for the tab character directly (sort of) by typing quote, then control-v, then another quote:
   

   $ grep "   " foo
   Hi there
   this is a       real tab

2. cat -T is an option to show the tabs in a file as "^I".
   It's tricky to grep for the "^I" though. grep \^I and grep "^I" don't work. You need grep '\^I'
   

   $ cat -T foo
   Hi there^I
   this is a ^Ireal tab
   isn't it.
   $ cat -T foo | grep "^I"
   $ cat -T foo | grep '\^I'
   Hi there^I
   this is a ^Ireal tab

vi split screen

:vsp

for vertical split screen.
Type control-W control-W to switch between them.

More at http://jmcpherson.org/windows.html

tar -z vs tar -j

tar -j uses bzip2
tar -z uses gzip

bzip2 compresses more but slower.

http://jeremy.zawodny.com/blog/archives/000953.html
http://en.wikipedia.org/wiki/Bzip2

tar -cvjf  new.tar dir_or_files

tar -xvjf new.tar

 Or should there be a "j" in the ".tar" so people know they need 'j' to extract?  Like: 

tar -cvzf new.tar.Z dir_or_files

tar -xzvf new.tar.Z

DOS scripting: nesting if's in place of AND

I want to say 'IF EXIST file1 AND EXIST file2'... but there's no AND.

But I can nest the if EXIST's:

if EXIST file1 (if EXIST file2 echo Both There)

For the ELSE, it gets more complicated:

if EXIST file1 (if EXIST file2 (echo Both There) ELSE echo Nope)

 Will echo Nope if file1 exists and not file 2, but not if file2 exists but not file1.  So you need:

if EXIST file1 (if EXIST file2 (echo Both There) ELSE echo Not 2) ELSE echo Not 1

 That will tell you either "Both There", or "Not 1" if only file1 isn't, or "Not 2" if only file2 isn't.

python: function with optional/default argument

>def testf(foo="blah"):
... print foo
...
>testf("hi there")
hi there
>testf()
blah

When Python shows you where the shared object is and says it doesn't exist

We had this error and it was driving me crazy:

    ImportError: [/.../..]/python2.4/lib/python2.4/lib-dynload/datetime.so: cannot open shared object file: No such file or directory
    [01,003839]PYTHON Error : Undefined Python Module :

When I did an ls for that directory, the shared object was in fact there.  It didn't even help to move the shared object into the same directory as my code.  I googled and looked on the python.org website and couldn't find any other mention of it.  That is, besides making sure the python path and load path were set and that the .so was executable, but that didn't solve the problem.

It turned out that the problem was that calls to the python module were embedded in a larger C program that had compiled and linked pointing to a 32-bit version of python2.4.  But the script that was trying to run the compiled code was pointing to a 64-bit version of python2.4.

Make sure all your python symbolic links are pointing to the same python.

To see if this is your problem, run the Linux "file" command on the .so and on the c-executable that's linked to the python module that's importing the .so.

 $file /[..]/python2.4/lib/python2.4/lib-dynload/datetime.so
/[..]/python2.4/lib/python2.4/lib-dynload/datetime.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped

$file cpgm
cpgm: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

To solve this, re-compile cpgm in 64 bits, or point to the 32-bit version of python to run it.

$ file cpgm
cpgm: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

You can get into this trouble on a 64-bit machine or VM since it will run 32-bit compiled code (whereas a 32-bit machine won't run 64-bit compiled code.)

python: time.strptime() instead of datetime.strptime()

Either one worked correctly from the python prompt, but when I embedded it in another tool,

fdate=time.strptime("SEPTEMBER 19 2010", "%B %d %Y")

worked, and

fdate=datetime.strptime("SEPTEMBER 19 2010", "%B %d %Y")

triggered an exception.

python: import

Here's a good link that explains how to use import.

If I have one file in the directory with only one function that I want to use in another file, I think it's clearer to say this at the top of the other python file:

from my_one_function_file import my_one_function

Then if I stop using that function, I know I can get rid of the import statement. When I look at the import statement, I know exactly why it is there.

The link recommends always using "import", which would mean:

import my_one_function_file

then calling my_one_function via

my_one_function_file.my_one_function()

In general, this recommendation does look like wise practice. It's more object-oriented.

python: mapping one list to another

I have two lists of strings. I want to associate the names of the first list with the values in the second.
Trying to do it with the mapping or dictionary type:

>>> x=['a','b']
>>> x
['a', 'b']
>>> y=['c','d']
>>> z={}
>>> z.fromkeys(x,y)
{'a': ['c', 'd'], 'b': ['c', 'd']}

That's not what I want..  Instead:

>>> d=dict(zip(x,y))
>>> d
{'a': 'c', 'b': 'd'}
>>>

That's as close as I can get it.

>>> d['a']
'c'
>>> d['b']
'd'

It seems awkward, but if you put it in a loop, it looks fine:

 >>> x=['a','b','c','d','e']
>>> y=['q','r','s','t','u']
>>> z=zip(x,y)
>>> z
[('a', 'q'), ('b', 'r'), ('c', 's'), ('d', 't'), ('e', 'u')]
>>> d=dict(z)
>>> d
{'a': 'q', 'c': 's', 'b': 'r', 'e': 'u', 'd': 't'}
>>> for key in x:
...   print key, "=", d[key]
...
a = q
b = r
c = s
d = t
e = u
>>>

python: looping through filenames in a directory

import os
...
dirname=os.getcwd()
ext_name=".txt"
...
try:
        for file in os.listdir(dirname):
            if file.endswith(ext_name):
                #do whatever
except:

python: setting two things at once, links for getopt, csv.reader,

The code I need to modify uses csv.reader and getopt

and it can set two values at once!

 >>> a,b="c","d"
>>> a
'c'
>>> b
'd'
>>> a,b='f g'.split()
>>> a
'f'
>>> b
'g'