Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regulator behavior after short circuit #135

Closed
whitequark opened this issue Apr 24, 2019 · 10 comments
Closed

Regulator behavior after short circuit #135

whitequark opened this issue Apr 24, 2019 · 10 comments
Labels
qualification Lifecycle stage: qualification revC Hardware revision: C

Comments

@whitequark
Copy link
Member

whitequark commented Apr 24, 2019

To reproduce this issue, do this:

  1. glasgow voltage A 5.0
  2. Short Vio to GND. Do this in a way that doesn't bounce as bounce will hide the issue.
  3. The board stays powered down--the 3V3 rail is dark because of brownout detection. However a considerable current is flowing through the regulator.
  4. Note that regulator EN pin is actually low.
@whitequark whitequark added revC Hardware revision: C qualification Lifecycle stage: qualification labels Apr 24, 2019
@marcan
Copy link
Member

marcan commented Apr 25, 2019

Okay, so the problem here is this.

We have two current limits: the one inherent to the TPS73101DBV and the upstream TPD3S014. The problem is that the latter trips faster and harder, and that the former has a memory characteristic.

  1. When Vio is shorted hard to GND, current begins flowing through both devices. The TPD3S014 detects this overload earlier, and starts limiting current. This stabilizes with the TPD3S014 dropping about 4.5V and the TPS73101DBV dropping less than 0.4V or thereabouts.
  2. At this point, the internal 5V rail is at 0.4V. This is not enough to power any logic, so the TPS73101DBV no longer functions.
  3. The 3.3V rail is also going low, and with it EN, but by this point the TPS73101DBV cannot respond to this low-EN condition and properly shut down; instead, the gate of its pass transistor remains charged and passing current, as it has no discharge resistor or other passive path from gate to GND when not powered properly.
  4. With the TPD3S014 dropping most of the voltage, it gets very hot; this causes it to go into thermal shutdown quickly.
  5. At this point the internal 5V rail drops to 0. However, the pass gate of the TPS73101DBV remains charged.
  6. After the thermal trip in the TPD3S014 resets a few milliseconds later, the TPS73101DBV begins passing current again since its gate is still charged. This causes another thermal trip, and the system oscillates through thermal trips and never recovers.

If the board is unplugged and plugged back in at this point, the lockup persists: the pass transistor gate of the TPS73101DBV seems to easily stay charged for over a minute while completely unpowered. Removing and reinstating the fault while unpowered has no effect; on next power-up the board still locks up. Glasgow "remembers" the lock-up state even when all power is removed.

To clear the condition, the board must be powered with no fault present (or the fault removed while powered). After that, with the regulator shut down, Vio can be shorted to GND again with no ill effects, and the board can be power cycled in this state normally.

Solutions being investigated:

  1. Use a regulator other than TPS73101DBV, which does not have the gate discharge problem.
  2. Replace the TPD3S014 with TPD3S044, which has a higher current limit and may allow enough voltage to remain on the 5V rail through a fault condition for the TPS73101DBV to cleanly shut itself down.
  3. Introduce a protection resistor in series with the TPD3S044 output. This can be before the feedback divider, so it would introduce dropout voltage, but not bad regulation. A Vio fault with a 1Ω resistance does not cause this problem; instead the TPS73101DBV drops most of the voltage and goes into thermal and current limiting, which is what we expect. Lower valued resistors to be tested.

@marcan
Copy link
Member

marcan commented Apr 25, 2019

255e8c5 implements solution #3. Waiting for parts to arrive to qualify it and determine final resistor value, and also test #2.

@sixtyfive
Copy link

sixtyfive commented Apr 25, 2019

Not sure if it has suitable specs for your purposes but I've had good experiences with the 3V3, PDIP version of the TPS71xx, which is also available as SO and TSSOP and with different fixed voltages or adjustable.

@whitequark
Copy link
Member Author

SO and TSSOP are far too large.

@marcan
Copy link
Member

marcan commented Apr 30, 2019

Okay, so the problem wasn't the TPD3S014 per se, at least not initially. The initial transient response where we lose 5V entirely is just due to lack of bulk capacitance on that rail (and things like impedance of the USB cable and transient response of whatever is upstream). After that, indeed the TPD3S014 current limit triggers ahead of the downstream regulator and we never recover.

Putting 150µF of capacitance on the internal 5V rail (after the TPD3S014) solves the problem entirely, with the 5V rail dropping briefly (below the brownout threshold, but not for long enough for the LM3880 to trigger a reset/shutdown sequence, and not low enough to brown out the 3V3 rail). This avoids a reset entirely, with the unit remaining active and functional through the event; the TPS73101DBV goes into current limiting and gets hot and that's it (this is the same thing that already happened if the output was enabled into a dead short; in that case the TPS73101DBV ramping up correctly ended up triggering the current limit without any upstream effects).

So we're going to go with several mitigations:

  1. Drop the EN pulldowns to 10K. This may or may not help anything, but it can't hurt and I feel that might help recovery over 100K.
  2. Put 0.68Ω in series with the regulator output, before the feedback divider. In my testing, this allows revC0 to recover from a Vio crowbar fault immediately; 5V rises high enough for the regulator to shut itself down. 0.68Ω at 300mA is 0.2V of dropout (this only matters when the output is set to 5V, as lower settings will still regulate properly).
    gl_5v0_crowbar_680mohm
    This resistance will be on the back of the board next to an empty footprint; users concerned about dropout at the 5V IO setting can either use a USB host that provides a slightly higher voltage (>5.2V under load will allow correct regulation at a 5V output setting when the load is high) or short out the resistor / install a 0 ohm resistor next to it.
  3. Add 150µF (tentative value, needs more testing; 220µF is sufficient with no series resistance, 150µF is sufficient with 0.68Ω resistance) across the internal 5V rail, after the TPD3S014. This solves the actual problem of a load transient browning out everything before the current limit kicks in.
    gl_5v0_crowbar_680mohm_150uF

(2) solves the steady-state problem (it lets Glasgow recover from a bad state where the TPD3S014 has gone into limiting mode and the output regulator is enabled), while (3) avoids getting into that state in the first place.

I don't think we need to switch to the TPD3S044 necessarily, but it would be an option if we want to allow higher current consumption, and it ought to allow us to use smaller protection resistors while still being able to get out of a lockup state.

Unrelatedly, we also found out (due to a PEBKAC error on my part while debugging this) that the LM3880 enable divider is wrong and brownout detection wasn't working; I'll fix this shortly.

Thoughts?

@marcan
Copy link
Member

marcan commented Apr 30, 2019

Looks like 100µF is enough; this droops the 3V3 rail down to 3.1V, but that should be acceptable for a glitch. Performance might be better with the production parts.

gl_5v0_crowbar_680mohm_100uF

I was able to shoehorn a 5mm SMT electrolytic into the layout:

image

marcan added a commit that referenced this issue Apr 30, 2019
@marcan marcan closed this as completed in 4500338 Apr 30, 2019
@marcan
Copy link
Member

marcan commented Apr 30, 2019

Turns out the TPD3S014 said 120µF was required for USB applications all along, derp. Bumped it to 150µF (which is about the highest we can get in that footprint), that should give us plenty of margin.

@electroniceel
Copy link
Member

I'm revisiting this issue as #165 will re-use the dropper resistor of point (2) above
as measurement shunt.

I've tested a stock revC1 with 0.68Ω and 150µF cap. Due to the capacitor,
there is no more brownout on the internal 5V when shorting one port rail.
But when shorting both ports (set to 5V) at once, there is still a brownout.

In stock revC1 this results in a shutdown of the FX2 and ICE40, recovery
after about 3 ms and properly disabled Vio.

I shorted the 0.68Ω resistor and created a short on both rails at once
using two AO3400A MOSFETs, controlled by a function generator.

DS4_QuickPrint15
DS4_QuickPrint14

Yellow: internal 5V rail
Magenta: Vio on Port A
Blue: Current flowing through Vio on Port A

So without the 0.68Ω resistor we now don't get the original issue, but
an oscillation at about 9 MHz over the time of a few milliseconds. This
shows that a resistor is definitely needed. The changed behavior could
have to do with the different test setup, like shorting both ports or
interaction of the LDO with the wire inductance of the test harness.

Next was a test with 0.22Ω, which shows a clear improvement, but still
not perfect behavior:

DS4_QuickPrint16
DS4_QuickPrint17
DS4_QuickPrint18

Increasing to 0.33Ω fixed these issues:
DS4_QuickPrint19
DS4_QuickPrint20

For implementing #165, it is planned to use a 0.47Ω resistor to give the
INA233 the wanted measurement range. 0.47Ω should give enough headroom
for process variation, different usb hosts and varying overload conditions, so that
the LDO stays stable under overload conditions.

@marcan
Copy link
Member

marcan commented Jun 8, 2020

Yikes, that oscillation. Worth noting that I never measured the Vio rail in the scope, just the internal 5V rail.

0.47Ω sounds reasonable; I forget how I came up with 0.68Ω, but I think I was leaving some margin (and also going with whatever values I had lying around to test with). Keep in mind that this was with a (probably higher ESR / lower perf) through hole cap tacked on. If 0.47Ω works then we can go with that and lower the dropout voltage.

That said, since the shunt is now a measurement device, we should consider a settings area to store the actual shunt value / calibration data somewhere in flash. This would allow for calibration, but also changing the value if, for example, a different manufacturing variant run (otherwise identially compatible with software) with different capacitors ends up requiring a higher value to avoid this issue.

@whitequark
Copy link
Member Author

we should consider a settings area to store the actual shunt value / calibration data somewhere in flash.

Reasonable, and in fact we can add this at any point when we need it in a backwards-compatible way (since I thought of this when designing the configuration area in the EEPROM). So there's no rush.

electroniceel added a commit that referenced this issue Jul 12, 2020
The original idea in #165 and #135 was to reuse the 0.47 Ohms dropper resistor
we already have for the regulator as current shunt too. As the INA233 has a fixed
measurement range of +- 81.9 mV, this would allow to measure 174 mA. My initial
thought was that this is enough, as the specced load of the TPS73101 is 150 mA.

Review and experimentation has shown that the TPS73101 can be used to 350 mA in
most configurations before limiting set in, for a programmed output of 5 V I
could use it even up to 420 mA. As one of the design goals of using the INA233
is overcurrent protection, the settable range should be higher than what the
regulator can output in best case.

So the design is changed to a 0.15 Ohms current shunt and an additional 0.33 Ohms
dropper resistor in series. This gives a total 0.48 Ohms drop for the regulator
and a measurable current range to 546 mA.

For both resistors a DNPed parallel footprint is provided as before, allowing
the user to reduce the voltage drop if desired.
esden pushed a commit that referenced this issue Aug 16, 2020
The original idea in #165 and #135 was to reuse the 0.47 Ohms dropper resistor
we already have for the regulator as current shunt too. As the INA233 has a fixed
measurement range of +- 81.9 mV, this would allow to measure 174 mA. My initial
thought was that this is enough, as the specced load of the TPS73101 is 150 mA.

Review and experimentation has shown that the TPS73101 can be used to 350 mA in
most configurations before limiting set in, for a programmed output of 5 V I
could use it even up to 420 mA. As one of the design goals of using the INA233
is overcurrent protection, the settable range should be higher than what the
regulator can output in best case.

So the design is changed to a 0.15 Ohms current shunt and an additional 0.33 Ohms
dropper resistor in series. This gives a total 0.48 Ohms drop for the regulator
and a measurable current range to 546 mA.

For both resistors a DNPed parallel footprint is provided as before, allowing
the user to reduce the voltage drop if desired.
whitequark pushed a commit that referenced this issue Dec 24, 2020
The original idea in #165 and #135 was to reuse the 0.47 Ohms dropper resistor
we already have for the regulator as current shunt too. As the INA233 has a fixed
measurement range of +- 81.9 mV, this would allow to measure 174 mA. My initial
thought was that this is enough, as the specced load of the TPS73101 is 150 mA.

Review and experimentation has shown that the TPS73101 can be used to 350 mA in
most configurations before limiting set in, for a programmed output of 5 V I
could use it even up to 420 mA. As one of the design goals of using the INA233
is overcurrent protection, the settable range should be higher than what the
regulator can output in best case.

So the design is changed to a 0.15 Ohms current shunt and an additional 0.33 Ohms
dropper resistor in series. This gives a total 0.48 Ohms drop for the regulator
and a measurable current range to 546 mA.

For both resistors a DNPed parallel footprint is provided as before, allowing
the user to reduce the voltage drop if desired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
qualification Lifecycle stage: qualification revC Hardware revision: C
Projects
None yet
Development

No branches or pull requests

4 participants