UNB/ CS/ David Bremner/ tags/ python

This feed contains pages with tag "python".

RSS Atom
CS2613 Assignment 4

Introduction

In this assignment you will write a class ColumnDict that takes a table in the form of a list of lists (e.g. from a CSV file) and provides a key-value data structure (dictionary) where the values are the columns of the table.

For full marks in this assignment your solutions should not use any loops, but only comprehensions (and other builtin functions as needed). Your solution should also work with arbitrarily large input.

  • Make sure you commit and push all your work using coursegit before 16:30 on Thursday November 27.

Dictionary functionality

Provide a constructor, and appropriate “dunder” methods so that the following tests pass. You should only store the data once.

def test_one_row():
    table= ColumnDict([["alice", "bob"],
                       [27, 32]])
    assert table["bob"] == [32]
    assert table["alice"] == [27]


def test_two_rows():
    table2= ColumnDict( [["alice", "bob"],
                         [1, 2],
                         [False, True]])
    assert table2["bob"] == [2,True]
    assert table2["alice"] == [1,False]

def test_three_cols():
    table3= ColumnDict( [["alice", "mallory", "bob"],
                         [1, 3, 2],
                         [False, None, True]])
    assert table3["bob"] == [2,True]
    assert table3["alice"] == [1,False]
    assert table3["mallory"] == [3,None]

def test_update():
      table3= ColumnDict( [["alice", "mallory", "bob"],
                           [1, 3, 2],
                           [False, None, True]])
      table3["bob"]=[42,False]
      assert table3["bob"] == [42,False]
      assert table3["alice"] == [1,False]
      assert table3["mallory"] == [3,None]

Selecting subsets of columns

Write a select method for you ColumnDict class that produces table (list of lists) table with subset of the columns. Your select method should handle any number of key arguments, including zero.

def test_select1():
   table3= ColumnDict( [["alice", "mallory", "bob"],
                        [1, 3, 2],
                        [False, None, True]])

   out=table3.select("alice", "mallory")
   assert out == [["alice", "mallory"],
                  [1, 3],
                  [False, None]]

def test_select2():
   table3= ColumnDict( [["alice", "mallory", "bob"],
                        [1, 3, 2],
                        [False, None, True]])

   assert table3.select() == []

Iterator

Provide appropriate dunder methods to support iteration. Do not copy or traverse the data more times than necessary. Do not use generators (i.e. yield). Your code should pass (at least) the following tests.

def test_iter1():
    rows= [["alice", "mallory", "bob"],
                         [1, 3, 2],
                         [False, None, True]]
    expected = [["alice", "bob", "mallory"],
                         [1, 2, 3],
                         [False, True, None]]
    table3= ColumnDict( rows )
    assert list(table3) == expected

def test_iter2():
    rows= [["alice", "mallory", "bob"],
                         [1, 3, 2],
                         [False, None, True]]
    expected = [["alice", "bob", "mallory"],
                         [1, 2, 3],
                         [False, True, None]]
    table3= ColumnDict( rows )
    out = []
    for row in table3:
        out.append(row)

    assert out == expected

def test_iter3():
    rows= [["alice", "mallory", "bob"],
           [1, 3, 2],
           [False, None, True]]
    expected = [["alice", "bob", "mallory"],
                [1, 2, 3],
                [False, True, None]]
    table3= ColumnDict( rows )
    assert [ row for row in table3 ] == expected
    assert list(table3) == expected
Posted Tags: /tags/python
Lab 20

Before the lab

Quiz

Study for the python quiz

Linear Algebra

One of the main features of Octave we will discuss is vectorization. To understand it, we need some background material on Linear Algebra. If you've taken a linear algebra course recently, this should be easy, otherwise you will probably need to review

Background

We will mainly rely on the Octave Interpreter Reference. A more tutorial style guide is the Gnu Octave Beginner's Guide, which is available in ebook form from the UNB library.

We'll refer to the online text by Robert Beezer for linear algebra background. We can happily ignore the stuff about complex numbers.

Python Quiz

The first half of the lab will be a quiz on python

Time
50 minutes
Activity
quiz

Running Octave

Time
5 minutes
Activity
Demo / discussion

Recursive Fibonacci

Time
10 minutes
Activity
Demo/Group programming.

Here is a JavaScript recursive function for Fibonacci:

function fib(n) {
    if (n<=0)
        return 0;
    if (n<=2)
        return 1;
    else
        return fib(n-1)+fib(n-2);
}

Let's translate this line by line into an Octave function.

Save the following in ~/cs2613/labs/L20/recfib.m; the name of the file must match the name of the function.

function ret = recfib(n)
  if (n <= 0)
    ret = 0;
  elseif (n <= 2)
    ret = 1;
  else
    ret = recfib(n-1) + recfib(n-2);
  endif
endfunction

Like the other programming languages we covered this term, there is a built in unit-test facility that we will use. Add the following to your function

%!assert (recfib(0) == 0);
%!assert (recfib(1) == 1);
%!assert (recfib(2) == 1);
%!assert (recfib(3) == 2);
%!assert (recfib(4) == 3);
%!assert (recfib(5) == 5);

Questions for your journal

Table based Fibonacci

Time
25 minutes
Activity
Programming puzzle

A powerful technique for making recursive A more problem specific approach (sometimes called dynamic programming) is to fill in values in a table.

Save the following in ~/cs2613/labs/L20/tabfib.m. Complete the missing line by comparing with the recursive version, and thinking about the array indexing.

function ret = tabfib(n)
  table = [0,1];
  for i = 3:(n+1)
    table(i)=
  endfor
  ret = table(n+1);
endfunction

%!assert (tabfib(0) == 0);
%!assert (tabfib(1) == 1);
%!assert (tabfib(2) == 1);
%!assert (tabfib(3) == 2);
%!assert (tabfib(4) == 3);
%!assert (tabfib(5) == 5);

Questions for your journal

Performance comparison

Time
10 minutes
Activity
Demo / discussion

Let's measure how much of a speedup we get by using a table.

Of course, the first rule of performance tuning is to carefully test any proposed improvement. The following code gives an extensible way to run simple timing tests, in a manner analogous to the Python timeit method, whose name it borrows.

# Based on an example from the Julia microbenchmark suite.

function timeit(func, argument, reps)
    times = zeros(reps, 1);

    for i=1:reps
      tic(); func(argument); times(i) = toc();
    end

    times = sort(times);
    fprintf ('%s\tmedian=%.3fms mean=%.3fms total=%.3fms\n',func2str(func), median(times)*1000,
             mean(times)*1000, sum(times)*1000);
endfunction

We can either use timeit from the octave command line, or build a little utility function like

function bench
  timeit(@recfib, 25, 10)
  timeit(@tabfib, 25, 10)
endfunction

Questions for your journal

Before next lab

Posted Tags: /tags/python
Lab 19

Before the lab

Read

Getting started

Generators and iterator classes

Time
25 minutes
Activity
individual

Consider the countdown generator from Section 6.2

def countdown(n):
    print('Counting down from', n)
    while n > 0:
        yield n
        n -= 1

Referring to Section 6.1 write an equivalent iterator class. Here we will implement the entire protocol with one class, but multiple classes like we did for JavaScript is also reasonable. Modify only the __next__ method of the following skeleton class so that given tests pass.

def countdown(n):
    yield 'starting'
    while n > 0:
        yield n
        n -= 1

def test_generator():
    counter = countdown(3)
    assert [ t for t in counter ] == \
        ['starting', 3,  2,  1]

class Counter:
    "Iterator class simulating countdown"
    def __init__(self,n):
        self.n = n
        self.first = True

    def __iter__(self):
        return self
    
    def __next__(self):
        # insert code here
        self.n -= 1
        return self.n+1

def test_class():
    counter = Counter(3)
    assert [ t for t in counter ] == \
        ['starting', 3,  2,  1]

What is __iter__ for?

Time
25 minutes
Activity
individual

Update the __iter__ method of your previous solution so that the following additional test passes

def test_twice():
    counter = Counter(3)
    assert [ t for t in counter ] == \
        ['starting', 3,  2,  1]
    assert [ t for t in counter ] == \
        ['starting', 3,  2,  1]

Using Generators I

Time
25 minutes
Activity
individual

Do Exercises 6.5 and 6.6.

Note that 6.5 just asks you to run some code, but you do have to be in the correct directory when running it.

For 6.6, you essentially need to split the given code so that the first part is the new generator follow.

Using Generators II

Time
25 minutes
Activity
individual

Start with the following stock related classes:

report.py

tableformat.py

stock.py

fileparse.py

portfolio.py

Complete Exercise 6.7. I suggest you make a new file follow2.py so that you preserve solutions for 6.6 and 6.7.

Posted Tags: /tags/python
JS Quiz / Lab 18

Before the Lab

Study for the JavaScript Quiz

Read

Quiz

Time
50 minutes
Activity
Quiz

The first half of the lab will be a javascript quiz

Getting started

Static methods

Time
25 minutes
Activity
Transform code, use template for static methods
    @staticmethod
    def read_portfolio(filename):
        # code from 4.3 goes here

Your completed static method should pass

def test_read_portfolio():
    portfolio = Stock.read_portfolio('Data/portfolio.csv')
    assert repr(portfolio[0:3]) == \
        "[Stock('AA', 100, 32.2), Stock('IBM', 50, 91.1), Stock('CAT', 150, 83.44)]"

Inheritance

Time
25 minutes
Activity
Refactor given code, work with class hierarchy

Start with following class hierarchy based on Exercises 4.5 and 4.6

class TableFormatter:
    def headings(self, headers):
        '''
        Emit the table headings.
        '''
        raise NotImplementedError()

    def row(self, rowdata):
        '''
        Emit a single row of table data.
        '''
        raise NotImplementedError()

class TextTableFormatter(TableFormatter):
    '''
    Emit a table in plain-text format
    '''
    def headings(self, headers):
        
        output = ''
        for h in headers:
            output += f'{h:>10s} '
        output+='\n'
        output+=(('-'*10 + ' ')*len(headers))
        output += '\n'
        return output
    
    def row(self, rowdata):
        output = ''
        for d in rowdata:
            output+=f'{d:>10s} '
        output += '\n'
        return output

def test_text_2():
    portfolio=stock.Stock.read_portfolio('Data/portfolio.csv')
    formatter= TextTableFormatter()
    output= formatter.headings(['Name','Shares','Price', 'Cost'])
    for obj in portfolio[0:3]:
        output +=formatter.row([obj.name,f'{obj.shares}',
                                f'{obj.price:0.2f}',f'{obj.cost():0.2f}'])

    assert '\n' + output == '''
      Name     Shares      Price       Cost 
---------- ---------- ---------- ---------- 
        AA        100      32.20    3220.00 
       IBM         50      91.10    4555.00 
       CAT        150      83.44   12516.00 
'''

 
def test_string_1():
    portfolio=stock.Stock.read_portfolio('Data/portfolio.csv')
    formatter= StockTableFormatter()
    output= formatter.headings(['Name','Shares','Price', 'Cost'])
    for obj in portfolio[0:3]:
        output +=formatter.row(obj)

    assert '\n' + output == '''
      Name     Shares      Price       Cost 
---------- ---------- ---------- ---------- 
        AA        100      32.20    3220.00 
       IBM         50      91.10    4555.00 
       CAT        150      83.44   12516.00 
'''

Before next lab

Read

Posted Tags: /tags/python
Lab 17

Before the Lab

Read

Getting started

Questions

Time
10 minutes
Activity
Add error check

Raising exceptions

Time
20 minutes
Activity
Add error check
def test_exception():
    with pytest.raises(RuntimeError) as e_info:
        prices = parse_csv('Data/prices.csv', select=['name','price'], has_headers=False)

Modules

Time
20 minutes
Activity
Refactor existing code to to reuse a function.
def test_prices():
    portfolio = read_portfolio('Data/portfolio.csv')

    prices = {  s['name']: s['price'] for s in portfolio }
    assert prices['AA'] == pytest.approx(32.2,abs=0.001)
    assert prices['GE'] == pytest.approx(40.37,abs=0.001)

Methods

Time
25 minutes
Activity
Write accessor and mutator methods.

def test_cost2():
    s = Stock('GOOG', 100, 490.10)
    assert s.cost() == pytest.approx(49010.0,0.001)

def test_sell():
    s = Stock('GOOG', 100, 490.10)
    s.sell(25)
    assert s.shares == 75
    assert s.cost() == pytest.approx(36757.5, 0.001)

Special Methods

Time
25 minutes
Activity
Code to test, learn about "dunder" methods.
def test_repr():
    goog = Stock('GOOG', 100, 490.1)
    assert repr(goog) == "Stock('GOOG', 100, 490.1)"

Before next lab

Study for the JavaScript Quiz

Read

Posted Tags: /tags/python
Lab 16

Before the Lab

Read

Getting Started

Sequences

Time
25 minutes
Activity
Convert REPL session to tests
def test_items():
    assert prices.items() == \
        [('GOOG', 490.1), ('AA', 23.45), ('IBM', 91.1), ('MSFT', 34.23)]

- Question for your journal: what is `list()` being used for in this section?
- Embed the following tests and make sure they pass

def test_zip():
    assert pricelist == \
        [(490.1, 'GOOG'), (23.45, 'AA'), (91.1, 'IBM'), (34.23, 'MSFT')]

def test_min_max():
    assert min(pricelist) == (23.45, 'AA')
    assert max(pricelist) == (490.1, 'GOOG')

List Comprehensions

Time
25 minutes
Activity
Write tests based on REPL session
import pytest
from report2 import read_portfolio
prices = {
        'GOOG' : 490.1,
        'AA' : 23.45,
        'CAT': 35.46,
        'IBM' : 91.1,
        'MSFT' : 34.23,
        'GE': 13.48,
    }

Note that the value test will need to be adjusted for a value of approximately 31167.10, since our price list is different from the one used in the book.

Higher order functions

Time
20 minutes
Activity
Assemble given pieces
def test_portfolio():
    portfolio = parse_csv('Data/portfolio.csv', types=[str, int, float])
    assert portfolio == [{'price': 32.2, 'name': 'AA', 'shares': 100},
                         {'price': 91.1, 'name': 'IBM', 'shares': 50},
                         {'price': 83.44, 'name': 'CAT', 'shares': 150},
                         {'price': 51.23, 'name': 'MSFT', 'shares': 200},
                         {'price': 40.37, 'name': 'GE', 'shares': 95},
                         {'price': 65.1, 'name': 'MSFT', 'shares': 50},
                         {'price': 70.44, 'name': 'IBM', 'shares': 100}]

def test_shares():
    shares_held = parse_csv('Data/portfolio.csv', select=['name', 'shares'], types=[str, int])
    assert shares_held == [{'name': 'AA', 'shares': 100}, {'name': 'IBM', 'shares': 50},
                           {'name': 'CAT', 'shares': 150}, {'name': 'MSFT', 'shares': 200},
                           {'name': 'GE', 'shares': 95}, {'name': 'MSFT', 'shares': 50},
                           {'name': 'IBM', 'shares': 100}]

Refactoring a function

Time
30 minutes
Activity
Refactor function to add feature
def test_tuple():
    prices = parse_csv('Data/prices.csv', types=[str,float], has_headers=False)
    assert prices == [('AA', 9.22), ('AXP', 24.85), ('BA', 44.85), ('BAC', 11.27),
                      ('C', 3.72), ('CAT', 35.46), ('CVX', 66.67), ('DD', 28.47),
                      ('DIS', 24.22), ('GE', 13.48), ('GM', 0.75), ('HD', 23.16),
                      ('HPQ', 34.35), ('IBM', 106.28), ('INTC', 15.72), ('JNJ', 55.16),
                      ('JPM', 36.9), ('KFT', 26.11), ('KO', 49.16), ('MCD', 58.99),
                      ('MMM', 57.1), ('MRK', 27.58), ('MSFT', 20.89), ('PFE', 15.19),
                      ('PG', 51.94), ('T', 24.79), ('UTX', 52.61), ('VZ', 29.26),
                      ('WMT', 49.74), ('XOM', 69.35)]
Posted Tags: /tags/python
Lab 15

Before the Lab

Read

Getting Started

Files

Time
20 minutes
Activity
Individual programming, synthesis

Data types

Tuples

Time
20 minutes
Activity
Individual programming, synthesis

Dictionaries

Time
20 minutes
Activity
Individual programming, synthesis
def test_cost():
    d = parse_row(row)
    cost = compute_cost (d)
    assert cost == pytest.approx(3220.000,abs=0.00000001)

def test_d():
    d = parse_row(row)
    update_dict(d)
    assert d == {'name': 'AA', 'shares': 75, 'price':32.2, 'date': (6, 11, 2007),
                 'account': 12345}

Containers: read_portfolio

Time
20 minutes
Activity
Write function based on template
def test_read():
    portfolio = read_portfolio('Data/portfolio.csv')
    assert portfolio == \
        [('AA', 100, 32.2), ('IBM', 50, 91.1),
         ('CAT', 150, 83.44), ('MSFT', 200, 51.23),
         ('GE', 95, 40.37), ('MSFT', 50, 65.1), ('IBM', 100, 70.44)]

def test_total():
    portfolio = read_portfolio('Data/portfolio.csv')
    total = 0.0
    for name, shares, price in portfolio:
        total += shares*price
    assert total == pytest.approx(44671.15,abs=0.001)
Time
20 minutes
Activity
Modify results of previous exercise
def test_read():
    portfolio = read_portfolio('Data/portfolio.csv')
    assert portfolio == \
        [{'name': 'AA', 'shares': 100, 'price': 32.2}, {'name': 'IBM', 'shares': 50, 'price': 91.1},
         {'name': 'CAT', 'shares': 150, 'price': 83.44}, {'name': 'MSFT', 'shares': 200, 'price': 51.23},
         {'name': 'GE', 'shares': 95, 'price': 40.37}, {'name': 'MSFT', 'shares': 50, 'price': 65.1},
         {'name': 'IBM', 'shares': 100, 'price': 70.44}]

def test_total():
    portfolio = read_portfolio('Data/portfolio.csv')

    total = 0.0
    for s in portfolio:
        total += s['shares']*s['price']
    assert total == pytest.approx(44671.15,abs=0.001)

Before next lab

Read

Posted Tags: /tags/python
Lab 14

Before the Lab

Questions

Time
5 Minutes
Activity
Group discussion

Getting Started

Mortgage Calculator

Time
20 Minutes
Activity
Modify given program

Do exercises 1.7 and 1.8 from Python Numbers

Introduction to Python strings

Time
20 Minutes
Activity
Experiment in Python REPL

Do exercises 1.13, 1.14, 1.15 from Python Strings

Pytest

Time
10 minutes
Activity
Demo
symbols = 'HPQ,AAPL,IBM,MSFT,YHOO,DOA,GOOG'
symlist = symbols.split(',')

def test_lookup0():
    assert symlist[0] == 'HPQ'

def test_lookup1():
    assert symlist[1] == 'AAPL'

[student@id414m22 L14]$ pytest listex.py
=================== test session starts ===================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home1/ugrad/student/cs2613/labs/L14
plugins: pylama-8.4.1, cov-4.1.0
collected 2 items

listex.py ..                                        [100%]

==================== 2 passed in 0.02s ====================

Lists and Pytest

Time
20 minutes
Activity
Individual programming from template

Functions and coverage

Time
20 minutes
Activity
Individual programming, modify previous solution

We have already been using python functions for pytest, without really thinking about how they work. In Part 1.7 of Practical Python, functions are explained.

def sumcount(n):
    '''
    Returns the sum of the first n integers
    '''
    total = 0
    while n > 0:
        total += n
        n -= 1
    return total

Before next lab

Read

Posted Tags: /tags/python
Indexing Debian's buildinfo

Introduction

Debian is currently collecting buildinfo but they are not very conveniently searchable. Eventually Chris Lamb's buildinfo.debian.net may solve this problem, but in the mean time, I decided to see how practical indexing the full set of buildinfo files is with sqlite.

Hack

  1. First you need a copy of the buildinfo files. This is currently about 2.6G, and unfortunately you need to be a debian developer to fetch it.

     $ rsync -avz mirror.ftp-master.debian.org:/srv/ftp-master.debian.org/buildinfo .

  2. Indexing takes about 15 minutes on my 5 year old machine (with an SSD). If you index all dependencies, you get a database of about 4G, probably because of my natural genius for database design. Restricting to debhelper and dh-elpa, it's about 17M.

     $ python3 index.py

    You need at least python3-debian installed

  3. Now you can do queries like

     $ sqlite3 depends.sqlite "select * from depends where depend='dh-elpa' and depend_version<='0106'"

    where 0106 is some adhoc normalization of 1.6

Conclusions

The version number hackery is pretty fragile, but good enough for my current purposes. A more serious limitation is that I don't currently have a nice (and you see how generous my definition of nice is) way of limiting to builds currently available e.g. in Debian unstable.

Posted Tags: /tags/python
Trivial example using python to hack ical

I could not find any nice examples of using the vobject class to filter an icalendar file. Here is what I got to work. I'm sure there is a nicer way. This strips all of the valarm subevents (reminders) from an icalendar file.

import vobject
import sys

cal=vobject.readOne(sys.stdin)

for ev in cal.vevent_list:
    if ev.contents.has_key(u'valarm'):
       del ev.contents[u'valarm']

print cal.serialize()
Posted Tags: /tags/python