Did a full discharge test over the weekend and this happened.

[kuhrd]

So I did a full discharge test of my batteries through the inverter running the entire house to see how much capacity in the battery bank was usable and also roughly how many days I could go with my normal use.  Started the test at about 7pm on Friday and the inverter shut off with a transformer voltage error on Sunday at about 5pm so I know I have a bit over a day and a half.  Maybe Sid can chime in with how low the voltage can go before it throws this error but I have the BMS set to kick out at 44V or 2.75V per cell but I have the inverter ATS/charging set to kick on at 48V (as it won't seem to let me set it higher) and my best guess when reading the BMS log is that the voltage dipped to 48.5 for a very brief time when the well pump started but then the inverter kicked out with this error.  I know that things can get dicey on the low end and I am not sure if the voltage dip was lower.  The BMS may not have recorded the lowest part of the dip and the inverter possibly tried to transfer and received an error when trying to do so under load.  But I have tested this manually before and it worked with a small load when I first hooked up the AC input on the inverter.

I powered the inverter off and back on again and it came up without issue and started the well pump and ran until the pressure switch satisfied and kicked out the pump.  Not sure if the batteries had enough time to recover in that minute or so it took me to go downstairs and reset the inverter.  I was keeping an eye on the battery voltage as I knew the battery would be getting close to the discharge knee but around 48v I should still have around 8-9% capacity left.  Just thought it was weird as I would have expected the inverter to transfer or shutdown due to low battery voltage and give a different error.  Either way I don't plan to run the battery below 50v under normal use with solar doing the bulk of the charging but wanted to see where the system would shutdown or if it would transfer properly and wasn't sure what to make of this.

[Sid Genetry Solar]

7 hours ago, kuhrd said:

but I have the inverter ATS/charging set to kick on at 48V (as it won't seem to let me set it higher)

Ah yes, the manual I need to write...

All of the battery voltage settings are referenced and scaled to the "nominal" setting.  For example, UVP settings can't be set higher than Nominal (normally 48v), and OVP settings can't be set less than Nominal.

Same with several of the trigger settings.  If you need a higher nominal ("center") voltage, you can increase that number in the configuration.  (Note that this will adjust ALL the other setpoints, as they're all referenced to this one.)

8 hours ago, kuhrd said:

Started the test at about 7pm on Friday and the inverter shut off with a transformer voltage error on Sunday at about 5pm so I know I have a bit over a day and a half.  Maybe Sid can chime in with how low the voltage can go before it throws this error

Umm...it shouldn't be throwing that error.  You should get a battery voltage error long before this error appears.

From one of your past posts, it sounds that you're running 1.1r2.  So there are only 2 cases that this error can appear:

1. Battery charge mode, and transformer charge current can't be regulated low enough
2. Inverter startup: while the soft-start ramp is still active, there is <64vAC detected on the output, AND the transformer throttle is >100%.  (This safety is to prevent blowing FETs with a miswired transformer/disconnected secondary/feedback failure.)  If you were running the inverter in power-save mode with the well pump...that could definitely cause a false trigger of this error.  (I probably need to revise the threshold down closer to 30v or less.)

At 48v, the transformer throttle will be past 100% for normal operation, and coupling that together with the well pump startup surge (in Power Save mode)--yeah, it could false trigger.  Need to address this in an update.

[kuhrd]

I believe I have power save mode disabled as the entire consumption of the house never drops below 45-50 watts anyway.  I will have to take another look at the settings.  I have the charge mode set to 50% IIRC but have tested it all the way up to 97% current without issue.  I will have to take a closer look at all of the settings when I get a chance.

[Sid Genetry Solar]

33 minutes ago, kuhrd said:

I believe I have power save mode disabled as the entire consumption of the house never drops below 45-50 watts anyway.  I will have to take another look at the settings.  I have the charge mode set to 50% IIRC but have tested it all the way up to 97% current without issue.  I will have to take a closer look at all of the settings when I get a chance.

OK, that's weird then, because that error can ONLY get triggered if the soft start is still active--and that lasts less than 1/4 second.  Unless there's a bug with the soft start flag (which I've seen...thought I fixed it though.  Will need to look into this.)

The error getting triggered at charge would be immediately after it switches over; this is in case the transformer voltage specification isn't matched up properly.  During charge mode, the MOS H-bridge works as a bridge rectifier in reverse (through the blocking diodes)--meaning that if the rectified transformer output voltage exceeds the battery voltage, the resulting charge current can't be controlled.  That's what this error is for; in a normal GS setup, the rectified transformer output voltage is far below the battery voltage, meaning that this error should not be possible to encounter.

 

Glad to hear that charge is working for you.  Currently, I've measured charge efficiency at ~60%; I plan to revise the charge technique to a more PFC-style method (via software update), which hopefully will increase the efficiency closer to 80%.  (Can't do anything about the voltage drop in the FETs!)

[Sid Genetry Solar]

Looked at the code that causes the "Xformer Volt Err", and turns out that I had a comparison backwards--which could cause this issue.  However, this error should only be possible during the soft-start when the inverter mode is beginning.  It can only happen later IF the soft-start flag somehow hasn't been cleared--but the code for that looks good.  Will need a bit more diagnostic info to say for sure--it helps a lot if you can readily reproduce the error.

[The Blind Wolf]

I did the same to my GS inverters this weekend, ran the batteries to zero, guess what, they didn't shut down till after the batteries were dead as a door.  I'm guessing my master battery settings got reset and just love sucking all that power out of those batts. 😛

 

 

[Sid Genetry Solar]

9 hours ago, The Blind Wolf said:

I did the same to my GS inverters this weekend, ran the batteries to zero, guess what, they didn't shut down till after the batteries were dead as a door.  I'm guessing my master battery settings got reset and just love sucking all that power out of those batts. 😛

 

 

Oops.  Yes, the settings in the master would have been reset; not sure if the slave inverter's settings were ever set.

[The Blind Wolf]

4 hours ago, Sid Genetry Solar said:

Oops.  Yes, the settings in the master would have been reset; not sure if the slave inverter's settings were ever set.

Nope we never set the slave, it wouldn't have matter if the master had it setting still, it would have shut the slave off.

 

Oh well, will get with you sometimes soon, not in a rush at the moment, got other things on my plate.  I did fix the wire heating issue though. got rid of any small wire deal.

[kuhrd]

On 6/29/2021 at 10:01 AM, Sid Genetry Solar said:

Looked at the code that causes the "Xformer Volt Err", and turns out that I had a comparison backwards--which could cause this issue.  However, this error should only be possible during the soft-start when the inverter mode is beginning.  It can only happen later IF the soft-start flag somehow hasn't been cleared--but the code for that looks good.  Will need a bit more diagnostic info to say for sure--it helps a lot if you can readily reproduce the error.

I haven't had the time to mess with it lately but I think I can reproduce it.  I am in the middle of helping my brother's family move homes and I just get back from a short 3 day vacation so the inverter has only been powering the minimum stuff lately.  I will try to get some more pictures or maybe even some video if it happens again.

[kuhrd]

Ok so I had some time to play around with this and I now know a bit more about what is happening.  So as early as 1.1r1 (IIRC) I had the GS inverter connected to grid and it seems to sync fine and even at least with the last revision 1.1r2 would allow me to charge or set to ATS and have it transfer successfully at least manually.  I had the inverter go into the same transformer voltage error when it tried to transfer and start charging the batteries at 50V on the bank.  Every time it tries to start charging the bank it gives this error it seems even though in the past I was able to manually transfer without issue and then manually charge the battery bank.  Not sure what has changed other than firmware since that time and I also would get the same error every once in a while with 1.1r2.  It seems I can pretty much reproduce the error at will.

The good news is that I can just set the inverter to inverter mode only or just keep the AC input breaker off and the inverter keeps running without issue. So it's not that the inverter was tripping on overload, its that the battery bank would dip down enough when starting the well pump that it would trigger the inverter to transfer and try to start charging the bank which was causing the error.

[Sid Genetry Solar]

Very good info to know. 

1.1r2 I wrote code to auto-zero the transformer amperage sensor at startup (previously it was a hardcode value).  Unfortunately that doesn't seem to be the "Holy Grail" software solution for the L1 filter cap being on the wrong side of said sensor.  If 1.1r1 worked fine and didn't throw the Xformer Polarity error, this could potentially explain why.

 

8 hours ago, kuhrd said:

So it's not that the inverter was tripping on overload, its that the battery bank would dip down enough when starting the well pump that it would trigger the inverter to transfer and try to start charging the bank which was causing the error.

Very good to know.

Looks like I need to increase the threshold for the Xformer Polarity--because if it's backwards, a LOT of power is going to flow through the FETs (basically a dead short through the blocking diodes).  I had 'scope shots of this (in tests), but can't seem to find them right now for some reason.

The Xformer Volt error in charge is more or less not needed in GS inverters--more there for customers (down the road I guess!) who want to make their own inverter, and might use a higher voltage transformer specification...and need to be aware of the minimum voltage charge output.  In other words...probably should push that one up a tad as well.

@kuhrd1.1r4 or immediate emergency?

[kuhrd]

14 minutes ago, Sid Genetry Solar said:

@kuhrd1.1r4 or immediate emergency?

I can wait for 1.1r4.  As far as I recall ATS and charging at least manually was working ok in 1.1r1 and I am pretty sure I started to see this error starting with 1.1r2 but I don't remember for sure.  I don't think I tested the ATS or charging with 1.1r3 right after the update.  As it is right now it seems like the second it tries to transfer and go into charge mode it throws the error. 

[Sid Genetry Solar]

I know I asked someone for this diagnostic...both Sean and I desperately need to get a CRM system implemented so we stop losing track of customers (much to their frustration).  If it was you, then just tell me 😉...

(did a search...it was @JNock)

...if you can report back what the Load Level value is at several stages, that'd be very helpful.  (OUT -> Diagnostic -> scroll down to Load Level).  I'm curious what it's reading in the following scenarios:

• no load...should be pretty close to zero
• small load (800W perhaps) on the L1 circuit (or maybe try a load on L1 and then on L2)...it should register very positive on one, and not much or at all on the other.
• ATS pass-thru (no charge)...should be pretty close to zero, or slightly negative.

Will see if I can make adjustments in the code so it works, and then release 1.1r4 to test.

[kuhrd]

So tonight it was able to successfully transfer to AC Pass-Thru and and I was also able to manually trigger it to charge without it getting mad with the Xformer Voltage Error so I took some pictures so you have a bit more info about what is going on.  Battery bank was at 52.3 when I manually set it to AC Priority and also when I manually triggered it to start charging.  I organized them into folders to make it easy to see what I was doing for each set of images.  https://drive.google.com/drive/folders/1Dnc4UhFIO34Mi_-axaoedpBKQF18Cb2Z?usp=sharing

Hopefully that helps provide some insight as to what is going on.

[kuhrd]

Oh almost forgot, it was 1300 watts because a milkhouse heater on low was the only portable load I had handy over 200 watts I could find.

[Sid Genetry Solar]

Thanks for the screenshots...very valuable debug info!

Numbers all look reasonably within range--holding at 0 when in pass-thru mode makes it all the more strange/weird that it's tripping off.  I presume that the Xformer Volt error is usually an instantaneous trip as soon as the relay tries to switch on (i.e. "click-clack -> error")?

Only suspicious number is the Xformer Volt in the No Load test...that number should be ~32vAC, not 35.4vAC.  Unless the actual AC output is really 265vAC (= 240v / 32v = 7.5 * 35.4 = 265.5v) due to miscalibration?  Might want to measure the AC output sometime with a DMM.

[kuhrd]

4 minutes ago, Sid Genetry Solar said:

Numbers all look reasonably within range--holding at 0 when in pass-thru mode makes it all the more strange/weird that it's tripping off.  I presume that the Xformer Volt error is usually an instantaneous trip as soon as the relay tries to switch on (i.e. "click-clack -> error")?

Only suspicious number is the Xformer Volt in the No Load test...that number should be ~32vAC, not 35.4vAC.  Unless the actual AC output is really 265vAC (= 240v / 32v = 7.5 * 35.4 = 265.5v) due to miscalibration?  Might want to measure the AC output sometime with a DMM.

Of the around 20 or so times it has given me this error several of them for sure happened as soon as the inverter tried to switch over since it happened when I was watching it do so pretty much as soon as the relay clicked. 

Each time I upgrade the firmware I check the calibration against 3 different DMMs just to be sure since things get expensive fast if its too far out. After the very first firmware update I did from the initial shipped firmware 1.0r22 to 1.1r1 and it lost a bunch of the calibration settings, I make it a habit to check after every update before flipping the breaker back on.  I can check it again but I did check it a week ago and the voltage was reading about 241-242  I have found that I have to wait with calibration until the inverter is warmed back up since the displayed voltage won't read accurate till everything is at a constant temp but the actual output voltage as measured with a DMM doesn't change much.  The AC Line voltage from the grid is 246-248 every time I have ever tested it and I try to keep the inverter output close to that but usually a bit under.  I haven't had to adjust the AC output calibration since I upgraded to 1.1r1 as the actual measured output has stayed within 240-246v range this whole time.

What is really weird is that I was able to turn the AC input on and power up the inverter without it erroring in these tests but 2 nights ago it would shutdown with the error as soon as it booted up and said it was In-Sync on the screen and the relay would click (I assume it was trying to transfer and charge the battery bank).  The only difference between these two times was that the battery voltage was low enough <51.6v that the inverter attempted to transfer to the grid so it could charge the batteries 2 nights ago and charging was set to Auto. Last night the bank was at 52.3 which is above the ATS voltage setting with charging set to manual and it never tried to transfer as soon as it synced up to grid frequency after it was powered on.  Could it be that it is transferring to fast and it doesn't have enough time to get a stable reading?  That wouldn't explain why the error is happening when the bank goes low 51v when the inverter has been running all week just fine unless it is something to do with charging kicking in to fast after the transfer happens in this case as well.  

[Sid Genetry Solar]

8 hours ago, kuhrd said:

Each time I upgrade the firmware I check the calibration against 3 different DMMs just to be sure since things get expensive fast if its too far out. After the very first firmware update I did from the initial shipped firmware 1.0r22 to 1.1r1 and it lost a bunch of the calibration settings, I make it a habit to check after every update before flipping the breaker back on.

Yah, guess I didn't write that in the release notes...or they aren't in an easy-to-see location.  1.1r0+ was a major shift from all previous versions (both in how the settings are stored AND in communication protocol), so instead of copying the settings over, I replaced them with stock constants--which evidently didn't work in your case.  Shouldn't need to tweak those settings for quite some time I hope, so future updates shouldn't change it.

 

8 hours ago, kuhrd said:

I have found that I have to wait with calibration until the inverter is warmed back up since the displayed voltage won't read accurate till everything is at a constant temp but the actual output voltage as measured with a DMM doesn't change much.

So bummed I didn't realize that optoisolators have a terrible temperature coefficient...that's what's causing the inaccurate readings and/or voltage shift on the output.  Switched to signal transformers on Rev. C...perfectly linear and with no temp coefficient.

 

8 hours ago, kuhrd said:

What is really weird is that I was able to turn the AC input on and power up the inverter without it erroring in these tests but 2 nights ago it would shutdown with the error as soon as it booted up and said it was In-Sync on the screen and the relay would click (I assume it was trying to transfer and charge the battery bank).  The only difference between these two times was that the battery voltage was low enough <51.6v that the inverter attempted to transfer to the grid so it could charge the batteries 2 nights ago and charging was set to Auto. Last night the bank was at 52.3 which is above the ATS voltage setting with charging set to manual and it never tried to transfer as soon as it synced up to grid frequency after it was powered on.  Could it be that it is transferring to fast and it doesn't have enough time to get a stable reading?  That wouldn't explain why the error is happening when the bank goes low 51v when the inverter has been running all week just fine unless it is something to do with charging kicking in to fast after the transfer happens in this case as well.  

I have a suspicion I know what the problem is...and I don't like it.  I believe you were one of the customers paying extra for express shipping...and while I thought we checked and redid the transformer spec on all of those units (gotta love the manufacturer, "oh, we did not know the voltage specification was +/-0.25v on the transformer"), it seems...that one slipped by.  Basically, I specified 8 turns on each low-voltage primary, but they did 8 on 1, and 9 on the others.  This is causing higher-than-expected output voltage for battery charge.  (We actually had them pull the first shipment of 25 inverters back from the shipping dock to fix the transformers after discovering this!)  Let me explain with mathematics:

Transformer spec for the current GS inverters should be 32v in, 240v out...clean ratio of 7.5.  But unfortunately it appears that your transformer is running closer to 35v in for 240v out...kinda right in line with what I recall measuring at Sean's place this past Dec. when we got them in.  This is not a critical problem, BUT it does cause several mathematical issues.

For one, the minimum DC input voltage for a true pure sine wave output at no load is...35.438 * sqrt(2) == 50.1v.  Not quite the 32 * sqrt(2) = 45.25v I'd designed for...but while it may be slightly annoying, it's not hurting anything.

Things get a tad trickier for AC input, and I believe also fully explains the "Xformer Volt" error you get when the battery is <51v.  If your grid is running ~248v in, we can calculate the transformer ratio at 240 / 35.438 = 6.77.  Thusly, 248v / 6.77 = 36.619vAC out from the transformer when the grid relay switches on.  And...36.619 * sqrt(2) == 51.78v MINIMUM DC OUTPUT IN CHARGE.  In other words, with AC Mains power, the inverter cannot regulate the battery charge less than 51.78v.  (Spec 7.5x ratio would output 33vAC @ 248v input = 46.76vDC minimum.)

So how does this cause the Xformer Volt error?  Well, if your batteries are < 51.78v, the instant the grid relay switches on, battery charge will start--despite the inverter doing nothing to regulate it.  While this won't be a problem (as the current will be relatively low), the inverter can see this "negative transformer current" despite charge not yet being engaged--and throws the Xformer Volt error, which I put in the code for just this purpose.

For 1.1r4, I can bump the error threshold up a tad for your usecase.  If the inverter's working fine otherwise, I can adjust the threshold.

[kuhrd]

That seems to make a lot more sense and fit with what I have been seeing.

Would this explain why the inverter seemed to struggle to start loads on occasion (starting the well pump) when the bank was low like <50Vdc and the flicker in the lights with the AC running?  Since a squareish wave is going to a bit harder to drive motors?

I can set the ATS setting at 52v (which would be around 30% SoC for the bank) and see if the error completely goes away when I let the bank run down and see if it transfers successfully.  This would maybe help verify if that fixes the issue.

If the transformer is slightly out of spec, is it possible to code in that setting change in the firmware specific to my my inverter's serial number so it doesn't affect your other inverters that are in the field presuming they are all running with the correct transformer specification?

I can obviously live with it as it is since it seems to be working properly other than that annoying issue.  Whenever the Rev C boards are available, and everything looks good or even when the 12 K is available (as I plan to purchase one eventually) is it possible to send the inverter back and have the transformer issue checked and corrected if possible and while it's in the shop pay to have the new Rev C board fitted?  I live in northern Wisconsin near Antigo so I might even be able to make a trip over to lower Michigan to save the shipping costs (and potential damage) if Sean is ok with it when that time comes.  Then if the 12K units are available at that time I would have one prepaid and waiting to be picked up at the same time.  I realize that this might be a year off depending on how things go.

[Sid Genetry Solar]

11 minutes ago, kuhrd said:

Would this explain why the inverter seemed to struggle to start loads on occasion (starting the well pump) when the bank was low like <50Vdc and the flicker in the lights with the AC running?  Since a squareish wave is going to a bit harder to drive motors?

Yes, it would...as once the inverter runs out of headroom, it has to square-up the wave, but that only works so far.

Flicker in the lights would be voltage regulation...shouldn't be doing that with the latest firmware...??

12 minutes ago, kuhrd said:

I can set the ATS setting at 52v (which would be around 30% SoC for the bank) and see if the error completely goes away when I let the bank run down and see if it transfers successfully.  This would maybe help verify if that fixes the issue.

Sure, that'd be a good way to check.

 

13 minutes ago, kuhrd said:

If the transformer is slightly out of spec, is it possible to code in that setting change in the firmware specific to my my inverter's serial number so it doesn't affect your other inverters that are in the field presuming they are all running with the correct transformer specification?

Not easily at all, I'm afraid.

 

13 minutes ago, kuhrd said:

is it possible to send the inverter back and have the transformer issue checked and corrected if possible

It can most definitely be corrected...using a variac to test the transformer, remove the extra turn on each of the affected primaries, and then cut the excess wire off...then remove the enamel from the (new) ends of the transformer strands....followed by crimping new ends on.  Didn't say it was fun though 😉.

 

14 minutes ago, kuhrd said:

and while it's in the shop pay to have the new Rev C board fitted?

Umm...not easily.  About all you would notice with Rev. C would be things like improved AC input relays (dual-pole main relay, and auto AC input voltage selection.)  The basic design and functional concept of the Rev. C board is identical to Rev. A.1.  Yes, there's a few new sensors added (battery current [for FET overload, not necessarily calibrated], AC input current)...but that's about it.

Rev. C control board will require the GS mainboard / MOS boards (changed pinout of the cable), and a new specialized WiFi board (changed feedback circuitry due to the inability to source the CS5463 power meter chip, necessitating switching to a different chip altogether)--so basically everything except the transformer and chassis.  Speaking of which...drilling new holes for the GS mainboard in the chassis is NOT easy.  That stainless steel case is extremely difficult to cut or drill; I admire the laser cutter that the manufacturer has! 

[kuhrd]

11 minutes ago, Sid Genetry Solar said:

Yes, it would...as once the inverter runs out of headroom, it has to square-up the wave, but that only works so far.

Flicker in the lights would be voltage regulation...shouldn't be doing that with the latest firmware...??

Sure, that'd be a good way to check.

 

Not easily at all, I'm afraid.

 

It can most definitely be corrected...using a variac to test the transformer, remove the extra turn on each of the affected primaries, and then cut the excess wire off...then remove the enamel from the (new) ends of the transformer strands....followed by crimping new ends on.  Didn't say it was fun though 😉.

 

Umm...not easily.  About all you would notice with Rev. C would be things like improved AC input relays (dual-pole main relay, and auto AC input voltage selection.)  The basic design and functional concept of the Rev. C board is identical to Rev. A.1.  Yes, there's a few new sensors added (battery current [for FET overload, not necessarily calibrated], AC input current)...but that's about it.

Rev. C control board will require the GS mainboard / MOS boards (changed pinout of the cable), and a new specialized WiFi board (changed feedback circuitry due to the inability to source the CS5463 power meter chip, necessitating switching to a different chip altogether)--so basically everything except the transformer and chassis.  Speaking of which...drilling new holes for the GS mainboard in the chassis is NOT easy.  That stainless steel case is extremely difficult to cut or drill; I admire the laser cutter that the manufacturer has! 

The flicker is nowhere near what it was in 1.1r2 but it is still present at times when inductive loads are running like the well pump and AC but it is almost not noticeable at this point.  The flicker could even be caused by the load that is running and the regulation setting in 1.1r2 was just making it very noticeable.  It's most noticeable in the few LED lights that don't have a capacitor but is also visible in incandescent bulbs like the one in the fridge.  But like I said the flicker is hardly noticeable and only when the large loads are running.

The biggest reasons I wanted to move to the Rev. C board was the voltage fluctuations with the temperature and the newer input relay since it switches both L1 and L2.  As far as the transformer modification, I guess I can wait for the warranty period to expire and then take a crack at disconnecting the transformer and testing it with a variac to get the turns right.  It wouldn't be the first time I have made adjustments to the number of turns on a transformer.  The other, less sophisticated option is to go to 17S on the pack so the battery will never go below 51.7 in normal operation but I would need a different BMS at that point.

I plan to keep the 6K installed as a backup (maybe even a slave to the 12K) when the 12K is available.  I will definitely need to add 20-30KWh of battery storage before that happens heh.

[Sid Genetry Solar]

30 minutes ago, kuhrd said:

The biggest reasons I wanted to move to the Rev. C board was the voltage fluctuations with the temperature and the newer input relay since it switches both L1 and L2. 

Yes, it would fix both I guess.

Technically, swapping the pinout on the current PJ mainboard would suffice to use a Rev. C board (though it would need a new WiFi board).

31 minutes ago, kuhrd said:

As far as the transformer modification, I guess I can wait for the warranty period to expire and then take a crack at disconnecting the transformer and testing it with a variac to get the turns right.  It wouldn't be the first time I have made adjustments to the number of turns on a transformer.

Will say it's aluminum wire, so it's a bit harder to work with ;-).  Since you're at 48v, the easiest solution would just be to pull a couple turns off one of the primaries to get it down to 32v...only need to redo one of the ends that way.