Sources of data

The links on the heading lead to the source location on the internet.

  • Registration and Exemption List
    • Available data
      • Region/State,
      • Station,
      • Unit-ID,
      • ROCPerMinute,
      • Registered Capacity,
      • Max Capacity
    • Problems
      • Quite a number of DUIDs are missing. Station name exists and RegCap, MaxCap are available.  However not SCADA Dispatch data for them so leave them out.
      • BOCORWF1 has duplicate entries, RegCap probably 113
  • Interconnector Capabilities
    For each interconnector

    • Registered capacity for each direction
  • Interconnectors
    Some background text on interconnectors
  • Region data for one day in 5 minute periods
    For each Region (State):

    • Demand,
    • Regional Reference Price,
    • Interconnector power,
    • Local Price for each generator if not zero.
  • Generator Dispatch (SCADA)
    For each generator:

    • dispatched power every 5 minutes.
  • CO2 Emission Factors aka Available Generators

    • Available Data
      • CO2-e Emission Factor,
      • Energy Source (a categorising property of generators).
    • Date
      • 2015-02-02
    • Problems:
      • Many DUIDs don’t match those from Registration and Exemption List when using column 6
        • AMGORGR1 2 and 3 apparently all refer to AMGORGR
        • APS1 refers to APS
        • BANKSPT2 might mean BANKSPT1
        • BBASEH_1, 2, 3  might mean BASEHOS which suggest using preceding column 5
      • Trying column 6 then column 5 if no match:
        • Lots more found in column 5
        • APD01, APPIN nothing similar
        • BERWICK, CLOVER could match by station name but R&E had no DUID so no generator
        • CHALLHWF no obvious match
      • Matching stations and assuming first
        • still 27 rows not matching
  • Existing scheduled and semi-scheduled generation
    Separate file per state. Duplicates data in Registration and Exemption List.
    NOT CURRENTLY PLANNED FOR INCORPORATION
  • Technical Parameters 
    • Summary
      • This includes interesting data.
      • The data is incomplete so no authoritative comparisons are possible using it.
      • The field which should identify the applicable generator is not always the DUID which is precise and unambiguous. 15 rows of data required assumptions to be made as to which generator they referred to.
    • Status
      Have been incorporated as far as possible. Of all the generators that were dispatched in the last month (2014-12), only the 21 listed below do not now have Technical Parameters.
    • Available data
      • The field Generation Type is potentially a categorising property, but mixes the Tech Type and Fuel Types of other categorising properties.
        Will be included if other properties depend on it.
      • Ramp rates up and down – better than ROCPerMinute from Registration and Exemption List.
      • Min Gen and Auxilary load appear not to be percentages but fractions, they are being multiplied by 100 and still called percentage.
      • Fuel Cost Profile leads to the Fuel Costs sheet of this spreadsheet.  The data for 2014-15 has been used.
    • Problems
      • The Plant column does not contain the same DUIDs as other sources. Only 114 row match existing generators and 207 do not match.
        For each generator.

        • There are duplicate DUIDs in Registration and Exemption List  that were not resolved. These have been resolve as explained below.
          • The rows with DUID of  ‘-‘ have been discarded
          • The rows with RegCap of  ‘-‘ or a MaxCap of  ‘-‘  have been discarded as they usually show data about physical units.
          • DUID: BOCORWF1 has two conflicting entries. The second has been used.
          • DUID: GRANGEAV has two conflicting entries. The second has been used.
          • DUID: INVICTA has three identical entries – not a real problem.
        • Eyeballing the lists of DUIDs from both sources indicates that application of a few simple rule could improve matters.
        • After a 1 month load of dispatches, which created some units not in Registrations and Exemptions, there were 223 units that dispatched power. Of these 36 were without Technical Parameters.
          • Apparently missing from  Technical Parameters (21)
            • BLUFF1
            • BOCORWF1
            • BUTLERSG
            • CLOVER
            • CLUNY
            • GB01
            • GERMCK
            • GULLRWF1
            • MBAHNTH
            • MERCER01
            • MLWF1
            • MUSSELR1
            • NBHWF1
            • PALOONA
            • PIONEER
            • REPULSE
            • ROWALLAN
            • RPCG
            • SHPUMP
            • SNOWYP
            • TARALGA1
          • Could be assumed to be in TPs (15). These assumptions have been incorporated.
            • CALL_A_4 could be CALL_B_1 and 2, and CPP_4. Is it CPP_4?
            • CHALLHWF is probably Challicum Hills Wind Farm
            • CNUNDAWF is probably Canunda
            • LEM_WIL is probably LM111-1
            • LI_WY_CA is probably LI111-1 or 2 or 3
            • MCKAY1 could be any or all of MCKAY11 to 16
            • PORTWF  could be Portland 2 or 3
            • PUMP1  could be W/HOE1 or W/HOE-1
            • PUMP2 could be W/HOE2 or W/HOE-2
            • SHGEN could be any or all of SHGEN01 to 04
            • SNOWNTH1 could be Snowtown Wind Farm
            • SNOWSTH1 could be Snowtown Wind Farm
            • SNOWTWN1 could be Snowtown Wind Farm
            • TVPP104  could be Tamar Valley CCGT or OCGT
            • WOOLNTH1  could be Woolnorth
          • There are quite a few units that have FuelPrimary or FuelDescriptor from which a TP_GenType could be derived unambiguously. The other missing ones have been set to Unspecified
      • Fuel costs are missing on the following Fuel Cost Profiles which are used by generators that dispatched a significant amount of energy in 2014:
        • Pelican Point
        • Osborne
        • Mortlake OCGT
        • Newport
        • Ladbroke Grove
        • Bairnsdale
        • Quarantine
        • Uranquinty
        • Roma
        • Oakey
        • Colongra
        • Somerton
        • Tamar Valley
        • Jeeralang
        • Mintaro
        • Laverton North
        • Valley Power
        • Barcaldine
        • Dry Creek
        • Bell Bay Three
      • The following generators are missing Fuel Cost Profiles:
      • The generators without a Fuel Cost as listed in the previous two list, have been given assumed values of the average of the Fuel Costs of generators with the same Fuel Type, Gen Type and State.
      • These Ramp Rates are given for a station whereas most processing is done with individual generators. Might be possible to apply ratio to generators
  • Emission Intensity Values
    • Summary
      The data is currently attached to the station not the generator.
    • Date
      2014-04-11 ACIL Allen
    • Data
      • Using only sheet: Summary EI and using Scope 1+3
        • C – Station: to match to units
        • A – Aux
        • U – Proposed new Emissions intensity – as generated
        • X – Proposed new Emissions intensity – sent out
    • Problems
      • There are no DUIDs. Matching will have to be attempted by Station Name.
      • There are 206 stations in the   Registration and Exemption List . Using exact text matching of Station Names, 115 stations were matched, 19 did not match. After name substitution 125 matched and 9 did not match.
        Currently 15 stations that have dispatching generators do not have  Emission Intensity Values.
      • Name substitutions:  Listed name:  Assumed correct name,
        ‘Dry Creek’: ‘Dry Creek Gas Turbine Station’,
        ‘Lemonthyme_Wilmot’: ‘Lemonthyme / Wilmot Power Station’,
        ‘Liapootah_Wayatinah_Catagunya’:
        ‘Catagunya / Liapootah / Wayatinah Power Station’,
        ‘McKay Creek’: ‘Mackay Gas Turbine’,
        ‘Mintaro’: ‘Mintaro Gas Turbine Station’,
        ‘Roma’: ‘Roma Gas Turbine Station’,
        ‘Snowtown Wind Farm’: ‘Snowtown Wind Farm Units 1 And 47’,
        ‘Tamar Valley CCGT’: ‘Tamar Valley Combined Cycle Power Station’,
        ‘Tamar Valley OCGT’: ‘Tamar Valley Peaking Power Station’,
        ‘Woolnorth Bluff Point Wind Farm’:
        ‘Woolnorth Studland Bay / Bluff Point Wind Farm’,
      •  Stations with Emission Intensity Values but not in Registration and Exemption List . Currently discarded. (9)
        Canunda Wind Farm
        Challicum Hills Wind Farm
        Collinsville
        Munmorah
        Portland Wind Farm Stage 2
        Portland Wind Farm Stage 3
        Shoalhaven
        SRPS
        Taralga Wind Farm
      • Stations, Dispatched during the year but without EIVs (16)
        …. Broken Hill Gas Turbines
        …. Rocky Point Cogeneration Plant
        …. Butlers Gorge Power Station
        …. Cluny Power Station
        …. Paloona Power Station
        …. Repulse Power Station
        …. Rowallan Power Station
        …. Mortons Lane Wind Farm
        …. Bendeela / Kangaroo Valley Power Station
        …. German Creek Power Station
        …. Moranbah North Power Station
        …. The Bluff Wind Farm
        …. Invicta Sugar Mill
        …. Bogong / Mckay Power Station
        …. North Brown Hill Wind Farm
        …. Wivenhoe Power Station No. 1 Pump
  • Fuel and Technology Data .pdf
    Fuel and Technology Data .xlsx

    • Summary Extensive amount of relevant data
    • Not yet used

 

 

34 thoughts on “Sources of data”

  1. “Quite a number of DUIDs are missing. Station name exists and RegCap, MaxCap are available. However not SCADA Dispatch data for them so leave them out.”

    I don’t think you should leave them out. I don’t think it’s a good idea to ‘adjust’ the AEMO data. If you change data in one place the credibility of the data base is immediately suspect. Just leave it as is. The ones not dispatch me be being refurbished and come on line later. Leave it to AEMO to maintain the data.

    Like

    Reply

  2. “Generator Dispatch (SCADA)
    For each generator:
    dispatched power every 5 minutes”

    The name is misleading. SCADAValue is ‘as-generated’, not ‘sent-out’ power. It is the power from each generator before ‘Auxiliaries’ and line losses to the ‘Regional Reference Node’. Can You please either change the name to
    SCADA (as-generated)
    or add after the name
    (power ‘as-generated’, not ‘sent-out’

    Like

    Reply

  3. “CO2 Emission Factors
    Available Data
    CO2-e Emission Factor,”

    The title is confusing. It is inconsistent throughout AEMO data bases because it’s taking time to fix it. Emissions Factor on the AEMO is sytem is being used for both t CO2-e/MWh and t CO2-e/GJ

    However, t CO2-e/MWh is also being called Emissions intensity on the AEMO site and elsewhere and is the term used throughout the EU (to my knowledge).

    Can you please either change the titles (two places) to

    Emissions Intensity (t CO2-e/MWh) or add it in brackets.

    Like

    Reply

  4. “Existing scheduled and semi-scheduled generation
    Separate file per state. Duplicates data in Registration and Exemption List.
    NOT CURRENTLY PLANNED FOR INCORPORATION”

    Do you have these three codes (exactly as they are, not a variation of them or a combination)?:

    Technology Type
    Fuel Type
    Dispatch Type

    We need those three codes, as those are the ones familiar to most policy analysts.

    Liked by 1 person

    Reply

  5. CO2 Emission Factors
    Available Data
    CO2-e Emission Factor,
    Energy Source (a categorising property of generators).
    Problems:

    Peter, Can I suggest you write to AEMO (or NEMMCO) and ask them to assign the DUID’s to every individual generator title or description in every file you are using. Send them spreadsheets so they can do it easily. They should respond on Monday.

    Like

    Reply

  6. “Technical Parametersd
    The field Generation Type is potentially a categorising property, but mixes the Tech Type and Fuel Types of other categorising properties.
    Will be included if other properties depend on it.”

    There are many fields on this sheet I think are relevant. See my email on this. I think we should have them all in the data base and allow people to use them for filters.

    1. State
    1. NTNDP Zone
    2. Generation Type
    3. Fuel Cost Profile
    4. Approximate Year Commissioned
    5. Min Gen (% of capacity)
    6. Auxiliary load (%)
    7. FOM (2013-14 $/MW/year)
    8. VOM (2013-14 $/MWh sent-out)
    9. Fugitive Emission Factor (kg CO2e/GJ)
    10. Combustion Emission Factor (kg CO2e/GJ) (no data)
    11. Percentage of emissions captured (%)
    12. Thermal Efficiency (%)

    Essential are: 1, 5, 6, 9, 12
    Highly Desirable: 2, 4,
    Desirable: 3, 7, 8

    Like

    Reply

    1. All included. Will make NTNDP Zone into a new Categorizing Type as soon as I find out what NTNDP means

      Like

      Reply

      1. Peter, I don’t know if you are just making a comment or what. The data needs to be the same way as in the AEMO file. I.e.; a factor, not a percentage.
        It is the NEM zone, such as North Coast NSW. They are subsets of each state. We need this breakdown for analysing the regional variation effects of wind generation

        Like

      2. Peter, on second thoughts, I don;t mind whether you leave the heading as in AEMO file (i.e %) and make the data percent, or leave the data as in AEMO and change the column heading. Up to you. But generally, I’d say don’t change any data or delete any.

        Like

  7. “Min Gen and Auxilary load appear not to be percentages but fractions, they are being multiplied by 100 and still called percentage.”

    You are correct. It is a factor, not a percentage. Just remove the “%” from the code name

    “Auxilary load”
    Not “Auxilary load %”

    Like

    Reply

  8. Regarding the data problems, can I suggest you send an Excel file to AEMO with all the spreadsheets you want data added. Tell them which columns need data added or checked/corrected. Colour the ones to be added / checked / corrected.

    Different colours would be best.

    Like

    Reply

  9. Peter, I don’t know if you are just making a comment or what. The data needs to be the same way as in the AEMO file. I.e.; a factor, not a percentage.

    Like

    Reply

  10. It is the NEM zone, such as North Coast NSW. They are subsets of each state. We need this breakdown for analysing the regional variation effects of wind generation

    Like

    Reply

  11. Numeric Filters include a misleading name ‘CO2 Emission Factor’. But it is actually the ‘CO2-e Emissions Intensity’, not the ‘CO2 Emissions Factor’. To be clear, it is ‘CO2-e’ not ‘CO2’ and it is ‘Emissions Intensity’, not ‘Emissions Factor’.

    We also need the ‘Combustion Emissions Factor’ to the right of ‘Fugitive Emissions Factor’. “Combustion Emission Factor (kg CO2e/GJ) are provided in a separate spreadsheet on the 2014 Planning Assumptions webpage” (but I haven’t found it).

    The revised Auxiliary loads (in %) and revised Emissions Intensity both as-generated and sent-out (updated 23 May 2014) are here: Emission Intensity Values . However, the Emissions Intensity you got from the CODEII Available Generators file is the most up to date (10 Dec 2014), so we should continue to use that source for Emissions Intensity.
    The AEMO web page is here: http://www.aemo.com.au/Electricity/Planning/Related-Information/Planning-Assumptions

    Could you please include the link to the source files in column headings so it is easy to check which source is used for each column of data.

    Like

    Reply

    1. This has Emissions Intensity for both as generated and sent out. I would like both because as generated will be more accurate when multiplied by SCADA than estimate the Sent out electricity to multiply by CO2EII.

      Could you run a check to see if the sent-out is the same as CO2EII in the Available Generators file?

      Like

      Reply

  13. Re; Emissions Intensity Values:

    1. Can you send the list of problems to AEMO and ask for DUID’s for all of the stations, or ask them to match those that you are not sure about

    2. Can you check the ‘Emissions intensities sent-out’ from this source against the CO2EII in the Available Generators file. If there are any differences ask AEMO why and which is the correct source to use.

    3. If they say use CO2EII, then ask them to please provide the correct ‘Emissions Intensity as-generated’ for all all generator units (or all stations).

    4. Check if the ‘Auxiliaries Load’ you have is the same as the ‘revised Auxiliaries %’ from here Emission Intensity Values file. If not. Ask AEMO which is correct

    5. By the way, I agree to show Auxiliaries as a % not as factor. That is the way generally used.

    Like

    Reply

  14. I am having trouble understanding the tables under the ‘Tabular’ pull down label.

    There are three tables called Table 1 Should have different numbers for each table. And a title. And links to the source data for each column.

    Table 1: Dispatched – Technical parameters. Can you please include a link to the source data for each column.
    Some columns are missing that I’d expect to be included because they are on the Technical Parameter sheet from which most of this was drawn. Can you add the station name, unit number, State and NTNDP Zone.
    Station Plant State NTNDP Zone Generation Type

    The ‘Generation Type’ is not the code from the Technical Parameters sheet, which are: Black Coal, Brown Coal, CCGT, OCGT, Distillate, Hydro, Wind, Cogen. These are simple, clear and commonly used despite mixing generator type and fuel.

    What’s the difference between Table 1 Units dispatched and Table 1: Dispatched – TechnicalParameters?

    Gen Class “Gas” should be OCGT and CCGT, not gas

    On units dispatched what is ‘Power Average’? what does that mean/ Is it the nameplate power of the unit or is an average over some period? What period? That needs to be stated.

    What is Energy ‘MWh’ average? Over what period? You need to state that. Actually this is confusing and meaningless. Delete it.

    Power ‘Std Dev’? Over what period? I suggest delete this.

    Remove all the price stuff from this page which is for the Senate Select Committee into Wind Turbines. We really don’t want it here. It will just create confusion and diversions. Put all this on a separate page dedicated to pricing stuff.

    Like

    Reply

  15. 6 priorities:

    1. The table of 1 year or SCADA (i.e. MW as-generated), Auxiliaries, MW sent-out, interconnector imports and exports, t CO2-e/MWh, t CO2-e/h, t CO2-e

    2. Table 2 with all constants (that are relevant for the emissions analysis) on a page and IDs so they can be assigned to every generator unit that appears on the primary data tables (including a link to the source for the data in each column and an explanation of what it is – ‘hover over’ is great)

    3. The charts I specified in spreadsheet previously sent

    4. Instructions on how to use the AusGrid website to access tables, select data to show in tables and to make the charts I described

    5. Glossary of terms (I sent you two previously)

    6. Specify the format you suggest to send the data in

    Like

    Reply

  16. Example charts:

    Y X Type
    Demand (MW) date or time line
    Power (as-generated) date or time line
    Power (sent-out) date or time line
    Capacity Factor (as-generated) date or time line
    Ramp Rate (MW/min) date or time line
    Interconnector (MW) date or time line
    CO2-e Emissions Intensity (t CO2-e/MWh) date or time line
    CO2-e Emissions Rate (t CO2-e/h) date or time line
    CO2-e Emissions (t CO2-e) date or time line
    CO2-e Emissions Rate (t CO2-e/h) Demand (MW) X-Y
    CO2-e Emissions Intensity (t CO2-e/MWh) Power (sent-out) X-Y
    CO2-e Emissions Rate (t CO2-e/h) Power (sent-out) X-Y
    CO2-e Emissions (t CO2-e) Power (sent-out) X-Y

    How do I:

    1. Set the start and end dates and the interval for the date and time charts?

    2. Filter by state, NEM zone, Generator type, Fuel type, etc.?

    Like

    Reply

  17. Regarding the charts, instead of building each one yourself, could you instead provide the means for users to build their own charts. For example allow users to build charts by selecting from:

    • Chart type: X-Y, frequency histogram, cumulative frequency chart,
    • Chart type: dots, line, dots & lines, bar, area chart, cumulative, etc.
    • Axes: MW Demand, MW as-generated, MW sent-out, t CO2-e/MWh, t CO2-e/h, t CO2-e, MW export, MW import
    • Axes: maximum and minimum value
    • Axes: interval between labels and grid lines
    • Axes scale: linear, logarithmic
    • Series: multiple series on a chart (e.g. by state, or generator type, fuel)
    • Filters: e.g. by state, or generator type, fuel etc.

    Like

    Reply

  18. I am beginning to realise that building such a facility would be an enormous challenge and unlikely to be successful. There is so much knowledge required about the available data, what can be derived from lower level data etc. I am using the Python unit: MatPlotLib – infinitely flexible but time consuming to find how to use the uncommon flexibility. It’s flexibility comes from it’s programming interface. I doubt it would be possible to achieve anything like it with a declarative interface that I think you a suggesting.

    It would certainly be possible to take some first steps in that direction when time permits. Until then the only option is to tell me what is your highest priority chart.

    Like

    Reply

  19. OK, disappointing but if that’s how it is that’s what we have to deal with. If I suggest three charts, can the user then select filters from the list of available filters?

    Can you make a number of charts one above the other that all share the same x axis time scale? Can you allow the user to select the start and end points of the time scale so they can see a year, or month or day, etc?

    Priority charts are:
    1. Chart set 1 – X axis scale is time / date, (line chart):
    1.1. Total generation sent out (or demand)
    1.2. Wind generation
    1.3 Total minus wind
    1.4 t CO2-e/h

    2. Chart set 2 – X axis scale is time / date (area chart total to 100%))
    2.1 Proportions of generation by gen/fuel type as defined below ,
    2.2 Proportions of t CO2-e/h by gen/fuel type as defined below ,
    Gen / fuel types
    – Brown coal
    – Black coal
    – CCGT
    – OCGT
    – Hydro
    – Wind
    – Other renewables

    See figures 1 and 2 hers for examples: http://joewheatley.net/wp-content/uploads/2012/11/co2.pdf

    I’d also like Figure 5 if you can do it (in components).

    Like

    Reply

    1. Chart 2 Your Gen/FuelTypes are close to those from Technical Parameters, which I call TP_GenType. Normally this would be an optional categorisation on a chart but if Chart 2 turns out to be a special purpose chart and not a general one, you can have TP_GenType as the default. Otherwise Chart 2 possible

      Like

      Reply

      1. I am not quite sure what this means. The categorisation needs to be what is needed for the analyses and what is generally used.

        Like

      2. The generator types I suggested we need for this analysis align with the sort of categorisation used for these types of analysis.

        Brown coal has very low fuel costs so tends have low SRMC and be last to be displaced. It has the highest GHG emissions. The power stations are different technology to the black coal power stations. They need to be kept separate

        Black coal is more flexible than brown coals and is the one most suffering from wind power in the NEM. We need it as separate. We could divide off super critical, but that may be a later refinement as there is very little super critical in Australia yet.

        OCGT and CCGT are quite different and the distinction is important. CCGT tends to be intermediate load in Australia with much higher capacity factors than OCGT. OCGT is a peaker plants. CCGT also has significantly lower emissions intensity than OCGT.

        Engines are also peakers and different emissions intensity.

        Hydro is zero emissions and used mostly for peak and intermediate load in Australia.

        Wind is the technology we are investigating so we need it separate from other renewables.

        All other renewables can be lumped together.

        There may be changes later, but this would be a good place to start.

        Like

      1. See my previous explanations. We have to use the same type of generation for sent out and for CO2 emissions. Either sent-out or as generated.

        CO2EII is sent-out, but it has to be multiplied by MW sent out. Estimating MW sent out is a rough estimate and a bit of a fudge that is favourable to wind.

        SCADA is as generated. To convert to sent out you must subtract Auxiliaries. That gives an approximate value for as-generated.

        Alternatively, we can use as generated. That means using SCADA for MW and computing CO2EII as generated from Thermal Efficiency x CO2-e Emissions Factor as-generated.

        The emissions factors (sent-out and as-generated) are in the file of Scope 1 + 3 emissions factors.

        Like

      2. To clarify what I meant here: “We have to use the same type of generation for sent out and for CO2 emissions. Either sent-out or as generated.”

        We have to multiply either:
        MW sent-out by CO2-e Emissions Intensity sent-out
        Or
        MW as-generated by CO2-e Emissions Intensity as-generated

        Where:
        SCADA = MW as-generated
        CO2EII = Emissions Intensity sent-out
        MW sent-out = MW as-generated – Auxiliaries
        Emissions Intensity as-generated = Thermal Efficiency x Emissions Factor as generated

        Like

    3. List of gen/fuel types needs to include engines:

      en / fuel types
      – Brown coal
      – Black coal
      – CCGT
      – OCGT
      – engines
      – Hydro
      – Wind
      – Other renewables

      Like

      Reply

      1. If you don’t like the name Gen/fuel’ types’ would you prefer simply ‘Technology’

        Like

Leave a comment