BCMS: Data Archiving

This entry serves as the main documentation for BCMS Data Archiving. Upon completion, it should have the following benefits:

1. Quicker backups.
2. Quicker data transfers/uploads.
3. Faster report generation.
4. Faster queries.

Now for the policies:

1. Following the method used for archiving the artran table, we will also archive these tables alongside artran_history to a new database called "history":
a. consumer_bill - Monthly (leave the previous month's data) - indexed on account_no_id, bill_no, bill_no_id
b. consumer_bill_details - Monthly (leave the previous month's data) - indexed on bill_no_id
c. bill_detailed_summarized - Monthly (leave the previous month's data) - indexed on account_no_id, bill_no
d. meter_reading - Monthly (leave the previous month's data)? - indexed on meter_reading_id, account_no_id, billing_month, actual_billing_month
e. payment - Monthly (leave the previous month's data) - indexed on account_no_id, ornumber, payment_date+teller_id
f. payment_details - Monthly (leave the previous month's data) - indexed on payment_detail_id, payment_id
g. payment_details_vat - Monthly (leave the previous month's data) - indexed on id, payment_detail_id
h. sql_logs, transaction_logs - Monthly (purge) - indexed on user_name, date?

2. All historical tables should be MyISAM.

3. Convert bill_detailed_summarized to MyISAM.

4. Update reports.

5. Immediately make a backup of the history database after archiving.

BCMS: Sacrificial Lamb

I heavily sacrified myself yesterday...

We were in a meeting to resolve the issues of LC's (load centers) servicing consumers in multiple areas. This problem surfaced because we wanted to be able to track consumers using feeders and load centers they are connected to, but our old feeder records are already several-years outdated (long ago, there were only 7 feeders but now there are 14 or 15), and this entails the need to update (and correct mistakes in) consumer records.

Several issues (some only related) came up:

1. RAJE (outsourced meter readers) will have to identify consumers which are recorded in the wrong areas (just a side problem).

2. Using new feeder-area groupings will be chaotic on RAJE's part (scheduling, etc.)

3. To achieve a proper hierarchical structure (Feeder - Area -LC - Meter), all individual LC's will have to be confined to one area each... an ENORMOUS task, to say the least, for the Engineering Department.

So I asked Eng'r. Lapuz if he needs to subdivide Feeders into areas, and when he said NO, I proposed that we:

1. Retain old feeder-area assignments as Revenue Areas, not Engineering Areas. The old FEEDER numbers will be renamed REVENUE GROUP numbers.

2. The FRANCHISE code at the beginning of each consumer's account number will be replaced with the FEEDER code/number they are serviced by/connected to.

The benefits of this scheme are:

1. NO CHANGES in RAJE's scheduling. They don't need to do any additional work!

2. NO TRANSFERRING of meters from LC to LC. They don't need to do any additional work!

3. NO DRASTIC CHANGES in Billing Reports and other stuff that refer to the old FEEDER-AREA numbers. (Note: Will assign modifications of report files to Rico.)

4. NO UNNECESSARY waste of paper (reports that fit 7/8 pages don't need to become 15 pages because of summarization by FEEDER).

The disadvantages:

1. SUBSTANTIALLY MORE WORK FOR moi!!! Aaaaaarrrgggghhh.... :

a. I need to modify the program to refer to old feeder numbers as revenue groups (I need to educate employees on referring to them from now on as such).

b. I need to restructure some tables in the database, which means I need to make sure the changes occur simultaneously at LH and RBM (2 San Fernando payment centers). Florida can be modified the day after.

c. For (b) to occur, I will have to create a temporary database for practice, along with a new set of code, just in case I commit a major blunder and have to revert to the old code.

Notes on 1.b.:
(i) Looks like uploading data from LH and RBM doesn't touch the consumer_master and consumer_meter_info tables, so I won't have to do a synchronized alteration of the table structures.

(ii) consumer_master: feeder_id and meter_information_id0 are both int(10) unsigned. Change these to transformer_id and pole_id, respectively. these have matching types. The feeder_id should then be picked up from the transformer_lup table. transformer here means LC (?)

MySQL InnoDB tablespace reclamation

Been having "table is full" and disk full errors again, so I decided to prune the data.

The usual steps were taken, following the procedures in this site: http://www.saturn.in/gpl/mysql.html

But before following those steps, ...
1. I made sure my latest backup of the main database was complete.
a. copied bill_settings, datafile_settings, and system_settings tables to my own database.
2. I dropped the main database.
3. Followed the instructions mentioned above.
4. Restored the 3 copied tables from my database because they're essential to the restore utility program.
5. Used "restore utility.exe" program to restore the tables.

For this housecleaning instance, I also used "remove zero from artran" program to transfer the fully-paid bills to the artran history table (this took more than 3 hours!!!). While doing this, I made a temporary database on the other server and uploaded last night's backup of artran and artran_vat. I created a table from the still unpaid bills (select account_no_id,applytodocno,sum(docamt) amount,docdate from artran group by account_no_id,applytodocno having amount <> 0.00).

Then I "inner joined" the resulting table to artran_vat and selected matching vat data into an outfile. I'm curious, though, why artran_vat shrunk from 476MB to 5.5MB (??!!). Only 61898 records?!?! With an average of 5 VAT charges per bill, those are only about 12,000 bills unpaid! What the ?!?!

Anyway, I'm rechecking now.

Found the problem. Looks like either a file system or a MyISAM limit was hit when I uploaded the latest artran_vat data to the new server.

Now, from an erroneous 5.5MB outfile, I have a 17MB one. And this time I'm sure the data is complete. Heh.

Addendum: Need to erase the BINLOGs after uploading the data.

BCMS improvement, plus possible bug

First, the good news. DPP requested me to include the Suffix in the Search window when looking for a name in the Ledger module. I did it one better: I included the middle name, too.

Now for the bad news. Agnes complained that the Miscellaneous Report-Final VAT Report is inconsistent with the "FINAL VAT Detailed Collection report per pay type". Particularly with the Oct. 2, 2007 Final VAT remittance of Account # 32780 (168 Victoria Square). I was at a loss why the report was not seeing the 2 payment details for that single O.R., and had spent several minutes looking for the problem on the query statement when I noticed my Payment Editor that the bill_no_id fields for the 2 "invisible" details was too high (>7,000,000) particularly because the bill_no was for the month of August 2007 and the bill_no_id for the 3rd detail was only in the 2,200,000 range and that was for Sept 2007! So I merely corrected the bill_no_id and the bill_date (which contained the "current" date for that posting - 10/02/2007 - instead of the real bill_date which is 08/14/2007).

I'll have to do experiments on this to catch the bug. Low priority, though.

BCMS: Bill Printing

Finished modification of Bill Statement report form for 3 more PC's (Gina, Rose, DPP).

Spent several minutes for Rose's and DPP's (the longest to edit).

Took me almost half a day because of some other work that got in the way.

SM Bill Adjustment

For the third time, I had to assist Mike in billing SM Pampanga because there needs to be a certain sequence in printing their monthly bill. Some monthly adjustments are posted so that it will reflect on their current bill.

BCMS improvements and fixes

I've made a lot of improvements and a few fixes to BCMS since the last time I posted here. In fact, there has been a slew of minor updates that I've already lost track. :-(

Oct. 25, 2007:
Transaction Logging: Added more descriptive data to the transaction logging of some reports, particularly the Detailed Collection Reports. The need arose when Agnes complained that her "same day" printout of Obet's collection report summary was changed when she reprinted it about a week after. I had to dig deep into the transaction_logs to do some forensic investigation and concluded that she inadventently CHECKED the "Use existing VAT summary" checkbox so that Obet's last few receipts were not summarized, even if she denied having checked it when I mentioned it to her before my investigation.

Ledger Module:
Adjusted some object placements in the Ledger Module so that the PRINT button, which has been fixed to be a MOMENTARY switch several weeks ago, will not disappear when resizing the BCMS window. I also incorporated some changes that Rico needed to identify the exact payment type and A/R type in artran.

Meter Reading (manual input): I've also corrected the code to fetch a consumer's name to place a space between the firstname and the middle initial. Long overdue.

Street Lights Settings: I've incorporated the Street Lights Report into the module. It's "by individual account" only. I also created a table for the City/Municipal street lights so that I don't have to do it in a spreadsheet every time they re-count the bulbs per barangay.

Oct. 26, 2007:
Bill Cancellation (and Unbill): Finally!!! I managed to modify these processes to include the corresponding VAT and UC charges. This is a major accomplishment, administrative-wise.
I didn't realize that when I modified the "Unbill" process, I failed to add a criterion to the UC detection to include the Unbundled UC charges, so they were still being incorrectly posted.

Oct. 30, 2007:
Billing Statements: Printed out statements of accounts for 3 big customers for BIR-auditing purposes. CFC discovered that since we went "Unbundled" the TIN #'s for the EVAT were removed from the text, so I now have to reformat 4 reports on 4 separate computers (well, 3 now, as I've already finished Mike's copy). I hope Gina's PC will accept Mike's copy without need for adjustments since they have the same printer model.

Consumer Management: I let Gina watch me as I edited an account which she isn't supposed to be able to modify. I hope she gets the workaround.

I'll post more when I recall them.